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Abstract 


A Runway Incursion Prevention System (RIPS) has been designed under previous research to 
enhance airport surface operations situation awareness and provide cockpit alerts of potential runway 
conflict, during transport aircraft category operations, in order to prevent runway incidents while also 
improving operations capability. This study investigated an adaptation of RIPS for low-end general 
aviation operations using a fixed-based simulator at the National Aeronautics and Space Administration 
(NASA) Langley Research Center (LaRC). The purpose of the study was to evaluate modified RIPS 
aircraft-based incursion detection algorithms and associated alerting and airport surface display concepts 
for low-end general aviation operations. This paper gives an overview of the system, simulation study, 
and test results. 


Introduction 

Runway incursions are a serious aviation safety hazard, particularly for general aviation (GA) 
operations. According to the Federal Aviation Administration (FAA) (FAA, 2007), during the four year 
period from fiscal year (FY) 2003 through FY 2006, there were approximately 250 million aircraft 
operations and 1,306 runway incursions reported at United States towered airports - approximately 5.2 
runway incursions for every one million operations. General aviation was involved in 72 percent of these 
incursions but only 55 percent of the operations. Eighty-two percent of the most severe incursions (98 of 
120 incursions) involved at least one GA aircraft. Four incursions resulted in collisions, with one of these 
collisions involving GA aircraft. These statistics do not consider incidents that occur at non-towered 
airports. 

Current FAA initiatives are targeting reductions in the severity and rate of runway incursions by 
implementing a combination of technology, infrastructure, procedural, and training interventions (FAA, 
2007). These solutions include Airport Surface Detection Equipment Model 3 (ASDE-3) and ASDE 
Model X (ASDE-X) radar; Airport Movement Area Safety System (AMASS); multilateration systems; 
in-pavement loops; Runway Status Eights (RWSE); enhanced controller training; airport surface 
operations advisory circulars; improved airport markings; improved pilot education, training, and 
awareness; and revised pilot/controller communications phraseology. These efforts target improved 
awareness and enhanced surveillance, but do not include on-board technology solutions for the flight 
deck. 

The National Transportation Safety Board (NTSB) considers runway incursions to be a serious 
aviation safety hazard, listing runway incursion prevention as one of their “most wanted” transportation 
safety improvements (NTSB, 2007). The NTSB specifically recommends that the FAA implement 
technology that “give immediate warnings of probable collisions/incursions directly to flight crews in the 
cockpit” (NTSB, 2000). In response to this recommendation, the FAA has begun to research the concept 
of transmitting ground-generated incursion alerts to aircraft and vehicles. 

NASA developed a Runway Incursion Prevention System (RIPS) for commercial and business 
transport aircraft operations to improve airport safety by providing surface situation awareness (SA) 
information and guidance cues, and alerts of runway conflicts and route deviations directly to the flight 
crew. The system was evaluated in several flight tests and simulation studies (Jones, Quach, and Young, 
2001; Jones, 2002; Jones, 2005). 

In the present work, RIPS was adapted for small-aircraft, GA operations. A piloted simulation study 
was conducted at NASA EaRC to evaluate RIPS for low-end GA operations that focused on evaluation of 
incursion detection algorithms and cockpit display concepts. This paper presents an overview of the 
system, description of the simulation study, and test results. 
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System Description 


Simulation Facility 


Flight Deck Simulator 

NASA LaRC’s Integration Flight Deck (IFD), normally used as a transport-category, fixed-based, 
high-fidelity, flight simulator, was adapted for this study because its visual, tactile, and audio capabilities 
provided the highest level of fidelity at LaRC toward meeting the research objectives (e.g., visual traffic 
acquisition). The IFD was configured to emulate a Cessna 206 (C-206) GA aircraft (herein referred to as 
the ownship). A six-degree-of-ffeedom, non-linear, simulation model of the C-206 and representative 
control-force and braking models were used. The left throttle was utilized to control the aircraft power 
while the right throttle was utilized to control the fuel mixture. In order to avoid distractions, extraneous 
displays and gauges were covered or turned off during data collection. 

As shown in Figure 1, an electronic research display (RD) was installed on the instrument panel 
directly in front of the left seat and control yoke. The RD was composed of two 10.4. inch liquid crystal 
displays and was capable of displaying two separate digital displays, side-by-side. An electronic flight 
bag (EFB) display, located to the left of the RD, was used to present the airport surface map display 
concepts described below. This display measured 10.4 inches diagonally with a resolution of 1280 x 
1024 pixels. The collimated out-the-window scene provided a 200 degree horizontal by 40 degree 
vertical field of view at 26 pixels per degree resolution. 

This configuration was used in a previous simulation, and feedback from the subjects of that study 
indicated that the simulation fidelity and cockpit environment were sufficiently representative of a C-206 
aircraft (Bartolone et al, 2005). 

Traffic position data were “broadcast” at a 1 Hz rate. No additional latency or surveillance data 
inaccuracies were applied. Ownship position data were updated at 20 Hz. 



Figure 1. IFD simulator cockpit configuration and displays. 
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Air Traffic Control Simulation 

Approach and tower air traffic control (ATC) instructions and pilot requests and replies were used in 
the simulation to increase the simulation fidelity of the terminal area environment and provide normal 
pilot workload demands during the study. All ATC and other aircraft radio messages were pre-recorded 
using different voices. The messages were then played through the flight deck speaker system when the 
ownship and simulated traffic reached specified locations. The subject pilots were asked to provide radio 
replies, when requested by the pre-recorded ATC messages, as per normal operating procedures. 

Research Displays 

This simulation study was designed for low-end, GA aircraft; therefore, standard round dial 
instrumentation was used as the “baseline” display concept around which various airport surface map 
formats and alerting concepts were experimentally evaluated. The genesis and background for the RIPS 
concepts are described in detail from previous research (Jones, et al, 2001; Jones, 2002; and Jones, 2005). 
The map format and alerting concepts were intended to identify which RIPS elements from previous 
research, as well as new elements, are most applicable and necessary to prevent runway incursion 
accidents and incidents for low-end, GA aircraft operations. 

The round-dial displays were shown on the RD located in front of the evaluation pilot (EP) (see Flight 
Deck Simulator section). 

When dictated by experimental condition, an airport surface map was displayed on the EFB located to 
the left of the round dial display. The surface map was generated using a Reno/Tahoe International 
(KRNO) airport geographic database developed to RTCA standards (RTCA, 2001). The map scale was 
set to 2.5 nm for the airborne scenarios and 1.5 nm for the ground based scenarios and was not pilot- 
selectable. Audible alerts sounded through the flight deck speaker system. 

Seven display conditions were evaluated during the course of the study as described below. 

Baseline (B) 

The Baseline (B) display condition consisted of a set of six instruments (airspeed indicator, attitude 
indicator, altimeter, vertical speed indicator, directional gyro, and turn coordinator) plus manifold 
pressure and instrument landing system data. All instruments were three inches in diameter and 
configured on the RD as shown in Figure 2. 



Figure 2. Baseline display condition. 
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Baseline with Surface Map and Ownship (BMO) 

The BMO display condition consisted of the baseline round dials displayed on the RD with the 
addition of a plan view surface map displayed on the EFB (Figure 3). This version of the surface map 
displayed an airport layout along with ownship position. Neither traffic nor ATC instructions (such as 
assigned taxi route) were shown. Incursion alerting was not part of this condition. 

Baseline with Surface Map and Ownship and Traffic (BMOT) 

The BMOT display condition was equivalent to the BMO condition but with the addition of traffic on 
the surface map (Figure 4). It was assumed that traffic was reported by an Automatic Dependent 
Surveillance-Broadcast (ADS-B) system. All traffic positions were reported without any positional 
inaccuracy (albeit at only a 1 Hz update rate). No data drop-outs, false reporting, or un-equipped traffic 
were simulated. 

Traffic was displayed as dark blue chevrons when on the ground and cyan chevrons when airborne. 
The direction of travel was indicated by the pointed end of the chevron. A circular symbol was used 
when traffic was traveling less than six knots, since the direction of travel could not be reliably estimated 
from just the simulated broadcast of position. Incursion alerting was not part of this condition. 



Figure 3. BMO display condition. 


Figure 4. BMOT display condition. 


Baseline with Audible Incursion Alerts (BA) 

The BA display condition consisted of the baseline round dials displayed on the RD and audible 
runway incursion alerts that sounded over the flight deck speaker when a potential conflict was detected. 
The detection method used to generate the alert is described in detail in the Runway Incursion Alerting 
section. 
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Baseline with Audible Incursion Alerts, Surface Map and Ownship (BAMO) 

The BAMO display condition was equivalent to the BMO condition with the addition of audible 
runway incursion alerts. 

Baseline with Incursion Alerts, Surface Map and Ownship, and Traffic (BAMOT) 

The BAMOT display condition was equivalent to the BMOT condition with the addition of both 
audible and visual runway incursion alerts (Figure 5). The alerting displays and detection method are 
described in detail in the Runway Incursion Alerting section. 

Baseline with Perspective Surface Map (BRIPS) 

The BRIPS display condition consisted of the baseline round dials displayed on the RD with a 
perspective surface map displayed on the EFB (Figure 6). The map graphically depicted a perspective, 
track-up airport layout with current ownship and traffic locations and incursion alerts, which are the same 
information presented with the plan view BAMOT condition. ATC instructions, including the approved 
taxi route and hold short locations, were sent via a simulated data-link and automatically loaded and 
depicted as a magenta route on the surface map and shown as an alpha-numeric string. Audible incursion 
alerts were also sounded. 

Audible route deviation and crossing hold alerts were also generated. Route deviation alerts were 
generated if ownship left its assigned path during taxi. Crossing hold alerts were generated if ownship 
crossed a hold line without clearance. 



Figure 5. BAMOT display condition 
with warning alert. 



Figure 6. BRIPS display condition with 
caution alert. 


5 





Runway Incursion Alerting 


Two different incursion detection algorithms, Runway Safety Monitor and PathProx^'^, were 
evaluated during the simulation study. Both algorithms were originally developed for large commercial 
and business transport aircraft operations and were modified for the low-end GA application reported 
herein. 

Runway Safety Monitor 

The Runway Safety Monitor (RSM) incursion detection algorithm (Green, 2006) uses a generic 
approach for detecting and generating incursion alerts. The RSM monitors traffic that enters a three- 
dimensional virtual protection zone around the runway that is being used by ownship. Incursion detection 
is based on the operational state of ownship and traffic, as well as other criteria, including separation and 
closure rate, to avoid false alerts. Identification, position, and altitude data are used to track the traffic in 
the protection zone. Traffic data projections are calculated within RSM since, from flight test experience, 
reliable position updates are not received at consistent intervals. RSM generates a Warning alert, which 
occurs when a runway incursion is detected and evasive action is required to avoid a potential collision. 
Information provided with each alert includes identification of the incurring traffic and separation 
distance to potential conflict. 

Modifications were made to RSM for this study to enable incursion detection and alerting for low-end 
GA operations. Ownship parameters were defined for the specifications and dimensions of the C-206 
aircraft. Improvements and refinements were made to the RSM alerting criteria that consider the 
availability of data, aircraft characteristics, and specific separation requirements for GA. For example, 
different minimum separation distances and aircraft land/rollout distances typically apply for GA aircraft, 
representative of a C-206. Since C-206 aircraft may travel slower than larger commercial aircraft, the 
minimum alerting distance can be less. See Green, et al. 2009 for detailed specifications for RSM alerting 
criteria for both low-end GA and non-GA aircraft. 

PathProx™ 

The PathProx^’^ detection algorithm (Cassell et al, 2003) is designed to handle over 40 specific 
runway incursion scenarios. Alerts are issued based on the states of the ownship and traffic and on 
conditions including position, speed, and track angle. PathProx^’^ generates two types of alerts analogous 
to the Traffic Alert and Collision Avoidance System (TCAS) approach. A Caution alert informs the flight 
crew of a potential incursion or an incursion where the conflict does not yet require evasive action. The 
crew can take evasive action, however, at their discretion. PathProx^'^ also generates Warning alerts 
when immediate evasive action is required. Information provided with each alert includes identification 
of the incurring traffic, the associated runway, and separation distance between the traffic and ownship. 

Alerting Displays 

Incursion alerts could be presented to the flight crew visually on the surface map and/or audibly. The 
visual and audible alert phraseology were identical. The alert phrases were designed to provide 
descriptive information regarding the location of the conflicting traffic. It was postulated that providing 
more details about the location of the incurring traffic would provide additional situation awareness, 
particularly during the condition when only audible alerts were provided. The incursion alert phrases 
developed for this study are listed in Table 1. As shown in Figures 5 and 6, the textual forms of these 
alerts were presented on the surface map. Also, the traffic symbol representing the conflicting traffic was 
enlarged, changed color (yellow for Caution and red for Warning), and was highlighted by a target 
designator box. The identification tag was also highlighted. In the event that the incurring traffic symbol 
was not shown because of the display scale, a symbol was pegged on the edge of the display in the 
direction of the traffic on the perspective surface map only. The estimated distance to conflict (in feet) 
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was shown beneath the ownship symbol. An example of a Warning alert for a scenario where both 
ownship and traffic are departing on intersecting runways is shown in Figure 5. An example of a Caution 
alert for a scenario where ownship is on approach to a runway with traffic in position and hold for 
departure is shown in Figure 6. 

Table 1. Incursion Alert Phrases 

Warning, Traffic 34R 
Caution, Traffic 34R 
Warning, Traffic 25 
Caution, Traffic 25 
Warning, Traffic Departing 34R 
Caution, Traffic Departing 34R 
Warning, Traffic Departing 25 
Caution, Traffic Departing 25 
Warning, Traffic Approaching 34R 
Caution, Traffic Approaching 34R 
Warning, Traffic Approaching 25 
Caution, Traffic Approaching 25 


Test Method 

The testing was conducted in two phases, rare event and usability. 

The rare event phase was designed to evaluate pilot reaction to a runway incursion event with a given 
display condition and incursion detection algorithm without expectation on the part of the subject 
(Newman and Foyle, 2003; Foyle and Flooey, 2003). This effect was created by flying 18 runs under 
“nominal” (i.e., no incursions) followed by the last run in this phase being a runway incursion event. The 
evaluation pilot (EP) was not told that this last run concluded the experiment phase or that a runway 
incursion was planned. 

Following the “rare event” phase, a usability phase was conducted. 

Test Matrix 

The test matrix conditions are identified in Table 2. A full-factorial evaluation across 3 alerting 
conditions (no alerts, audible alerts, and audible plus visual alerts) and 4 display conditions (Baseline, 
plan view map, plan view map with traffic, and perspective map with traffic) was considered but rejected 
for practical and expediency reasons as follows: 

• Evaluating visual alerts in the baseline and baseline with plan view map (ownship only) 
conditions was rejected as not applicable (N/A) because the map did not contain any traffic 
information; therefore, the value of adding and testing visual alerting for these cases was 
considered to be small. 

• It was assumed that if traffic was provided on a surface map, both audible and visual alerts 
would be displayed; therefore, the conditions with audible-only alerts were not evaluated. 

• The experiment matrix then contains a 2x2 evaluation of the influence of a plan view map (on 
and off, with ownship position only) and audible alert (on and off). 
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• The effect of a surface map displaying traffic in addition to ownship was evaluated with no 
incursion alerts and with combined audible and visual alerts. 

• Finally, the perspective surface map displaying ownship and traffic along with ATC 
instructions (such as the approved taxi route and hold short locations) was considered an 
advanced condition; therefore, only the most advanced display alerting condition (audible and 
visual alerts) was evaluated with this condition. 


Table 2. Display Test Conditions 



No Alert 

Audible Alert 

Audible & 
Visual Alert 

Baseline (B) 

B 

BA 

N/A 

B + plan view map with ownship 

BMO 

BAMO 

N/A 

B + plan view map with ownship & traffie 

BMOT 

Did not evaluate 

BAMOT 

B + perspeetive map with ownship & traffie 

Did not evaluate 

Did not evaluate 

BRIPS 


Rare Event Testing Phase 

Four display conditions (BMO, BMOT, BAMO, BAMOT) were evaluated across subjects. The EP 
flew only one display condition for 1 9 runs during the rare event testing phase. 

These four display conditions were chosen to evaluate the effects of (a) displaying traffic without 
alerting (BMO and BMOT) and (b) with alerting (BAMO and BAMOT). The evaluation of these four 
display conditions allowed for direct comparison of the efficacy of alerting without the display of traffic. 
The rare event testing matrix further enabled direct evaluation of the contribution of display of traffic, 
both with and without alerting, to determine whether just showing traffic is sufficient for situation 
awareness and runway incursion avoidance or whether alerting is required. 

Eighteen trials were randomly flown that consisted of six different approaches in varying day time 
visual meteorological conditions (VMC) and instrument meteorological conditions (IMC) visibility 
conditions (see Appendix A). The intent of these trials was to provide sufficient variety and task demands 
to hide the rare event, a runway incursion. Pilots were unaware of the total number of trials to be 
conducted. The final trial in the block was the runway incursion event (Scenario 1 as described below) 
conducted in marginal VMC conditions (3 nm visibility and 1000 ft ceiling). Scenario 1 - the 
arrival/takeoff hold incursion - was selected because of its prevalence in runway incursion incident and 
accident statistics. The RSM incursion detection algorithm was used as the alert source. A wind profile 
was introduced to add workload to the tasks (see Appendix A). 

Usability Testing Phase 

Following the rare event phase, a usability study phase evaluated the effectiveness of the display 
conditions for runway incursion prevention. Each EP evaluated all five incursion scenarios with the 
seven display conditions (see Appendix B). The first trial for each scenario group used the baseline 
display condition. All of the trials that provided alerting used the RSM incursion detection algorithm as 
the alert source. In this study phase, the subjects had an expectation for the study intent (i.e., runway 
incursions), but they did not know the scenario before the initial trial. Another purpose of the usability 
study was to evaluate the RSM and PathProx^'^ incursion detection algorithms for GA operations. Each 
EP evaluated both algorithms using all five incursion scenarios and the BRIPS display condition. All 
trials in the usability study were conducted in marginal VMC conditions (3 nm visibility and 1000 ft 
ceiling) without winds. 



Runway Incursion Scenarios 


Five incursion scenarios were developed. Within each scenario, an incursion was staged by a blunder 
from one of the participating other aircraft. 

A traffic pattern was established to create reasonable traffic flow at KRNO (Figure 7). Alternating 
arrivals and departures were simulated using Runway 34R with interleaving departures on Runway 25. 
Traffic traveled through the intersection of Runways 25 and 34R every minute. The incurring traffic was 
interleaved into this traffic flow. 

Every effort was made to produce similar timing for the scenarios; however, a certain amount of 
variability in the timing was naturally introduced due to the maneuvering conducted by the EP (i.e., 
approach speed, taxi speed, etc.). 


Scenario 1 - arrival/takeoff hold 

This scenario began with ownship approaching Runway 34R for landing, 3 nm from the threshold at 
1010 ft above ground level (AGE), and at an indicated airspeed of 90 kt. The EP was cleared to land by 
tower. Another aircraft was stopped and holding at the 34R hold line nearest to the runway threshold. 

Scenario 1 tested the incursion situation where an aircraft moves into position and holds for takeoff 
even though another aircraft was approaching the same runway for a landing (Figure 8). The traffic for 
this scenario, instead of holding short of the active runway, taxied into position without clearance when 
the ownship was 2 nm from the threshold and held in that location awaiting takeoff clearance. 


«oox2a» 
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Figure 7. Traffic pattern. Figure 8. Scenario 1 configuration. 
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Scenario 2 - departure/intersection departure 

This scenario began with ownship taxiing on Taxiway C at 8 kt toward Runway 34R threshold, 
approximately 500 ft from the threshold. The EP was cleared to taxi to Runway 34R via Taxiway C and 
hold short of Runway 34R. The traffic was cleared to taxi to Runway 34R via Taxiway L, holding short 
of Runway 34R for an intersection take-off (Taxiway L is approximately 3000 ft from Runway 34R 
threshold.) Ownship was then cleared for takeoff. 

Scenario 2 tested the incursion situation where one aircraft is departing as another aircraft enters the 
runway for an intersection departure (Figure 9). The traffic for this scenario, instead of holding short of 
the active runway, taxied across the hold line and entered the runway without clearance once the ownship 
began its takeoff (i.e., on runway heading and traveling greater than 10 kt ground speed). 

Scenario 3 - arrival/departure 

This scenario began with ownship approaching Runway 34R for landing, 3nm from the threshold at 
1010 ft AGE, and at an indicated airspeed of 90 kt. Another aircraft was taxiing on Taxiway E near the 
Runway 25 hold line at the threshold. The other aircraft was cleared into position for departure on 
Runway 25. The ownship EP was then cleared to land on Runway 34R by tower. 

Scenario 3 tested the incursion situation where an aircraft was departing even though another aircraft 
was landing on an intersecting runway (Figure 10). The traffic for this scenario, instead of holding in 
position, began its takeoff on Runway 25 without clearance as the ownship crossed the threshold of 
Runway 34R. 



10 


Scenario 4 - departure/departure 

This scenario began with ownship taxiing on Taxiway C at 8 kt toward Runway 34R threshold, 
approximately 500 ft from the threshold. The EP was cleared to taxi to Runway 34R via Taxiway C and 
hold short of Runway 34R. Another aircraft was taxiing on Taxiway L near the Runway 25 hold line at 
the threshold. The other aircraft was cleared into position for departure on Runway 25. The ownship EP 
was then cleared for takeoff on Runway 34R. 

Scenario 4 tested the incursion situation where an aircraft was departing even though another aircraft 
was departing on an intersection runway (Figure 11). The traffic for this scenario, instead of holding in 
position, began its takeoff on Runway 25 without clearance as the ownship began its takeoff (i.e., on 
runway heading and traveling greater than 10 kt ground speed). 
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Figure 11. Scenario 4 configuration. Figure 12. Scenario 5 configuration. 


Scenario 5 - taxi crossing/departure 

This scenario began with ownship parked on the ramp at the Mercury Air Center (Fixed-Base 
Operator (FBO)) facing Taxiway E. Another aircraft was stopped on Taxiway E behind the Runway 25 
hold line at the runway threshold. The EP was cleared to taxi to Runway 34R via Taxiway C, cleared to 
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cross Runway 25, and hold short of Runway 34R. The other aircraft was cleared into position for 
departure on Runway 25 while the ownship taxied out of the ramp. 

Scenario 5 tested the incursion situation where an aircraft taxis across a runway even though another 
aircraft is taking-off from the same runway (Figure 12). The traffic for this scenario, instead of holding in 
position, began its takeoff on Runway 25 without clearance as the ownship crossed the centerline of 
Taxiway L. 


Procedure 

Prior to the rare event testing phase, each EP participated in a briefing and training session. The 
training did not reveal the focus of the experiment (runway incursion prevention). 

The EP received training on the incursion alerting system prior to the rare event testing only if the 
display condition evaluated included alerting. The EP was trained to abort if a warning alert was given 
during departure, go-around if a warning alert was given on approach, and stop if a warning alert was 
given during taxi. The EP was not required to take evasive action when a caution alert was issued. 

Before each trial, the EP was briefed on the run conditions, e.g. approach or departure, visibility, and 
displays available. The case order list is shown in Appendix C. 

For the usability study phase, the EP received training on the incursion alerting system prior to data 
collection. Before each trial, the EP was briefed on the run conditions, e.g. approach or departure, 
visibility, alerting system selected, and displays available. The EP was asked to continue the maneuver 
until a warning alert was received for evaluation purposes. The case order list is shown in Appendix D. 

The test runs were documented via audio, video, and digital data recordings, and post-run, post-block, 
and post-test questionnaires (Appendices E through J). 

Evaluation Pilots 

Sixteen GA pilots served as EPs. The EPs were selected to create four cross-sections of flying 
experience representative of the Part 91 pilot population: low-time (<400 hours) visual flight rules 
(VFR), high- time (>400 hours) VFR, low-time (<1000 hours) instrument-rated, and high- time (>1000 
hours) instrument-rated (see Table 3). 


Table 3. EP Experience 


Low Time VFR 
(< 400 Flours) 

Fligh Time VFR 
(> 400 Hours) 

Low Time Instrument- 
Rated (< 1000 Hours) 

High Time Instrument- 
Rated (> 1000 Hours) 

EP 

Flight 

Flours 

EP 

Flight 

Hours 

EP 

Flight 

Hours 

EP 

Flight 

Hours 

1 

80 

5 

415 

9 

590 

13 

4170 

2 

300 

6 

765 

10 

393 

14 

15000 

3 

175 

7 

450 

11 

1000 

15 

5365 

4 

160 

8 

401 

12 

510 

16 

3500 

Mean 

178.8 

Mean 

507.8 

Mean 

623.3 

Mean 

7008.8 

St. Dev. 

91.0 

St. Dev. 

172.7 

St. Dev. 

263.9 

St. Dev. 

5383.1 
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Results 

A summary of quantitative and qualitative results is presented for the rare event testing and usability 
study phases. All data are referenced from the center of gravity of the aircraft. All statistically significant 
results are presented at the p < 0.05 level unless stated otherwise. 

Rare Event Testing Phase 

Each EP flew 18 various approach tasks before being presented with the runway incursion event 
(Scenario 1). 

The FAA runway incursion severity rating (FAA, 2007), described below, was used to categorize the 
runway incursion incident results. The data were independently evaluated by a subject matter expert for 
classification. 

Category A - Separation decreases, extreme action taken to narrowly avoid collision, or collision 
occurs; 

Category B - Separation decreases, significant potential for collision; 

Category C - Separation decreases, ample time and distance to avoid collision; 

Category D - Tittle or no chance of collision but meets definition of runway incursion. 

Fourteen of the 16 runway incursions resulted in the less hazardous Category C and D incursions, one 
resulted in a Category A incursion, and one resulted in a Category B incursion. (All trials generated at 
least a Category D rating because the scenario was designed to elicit a runway incursion situation.) 

The 14 less hazardous Category C and D incursions were mitigated by the EPs by conducting a go- 
around and gaining separation from the traffic. 

The Category A runway incursion occurred with the EP flying the BMOT display concept. Despite 
the traffic indications on the surface map and out-the-window visuals, the EP demonstrated no awareness 
of the runway traffic and over- flew the traffic and landed. 

The Category B incident occurred when the EP over- flew the runway traffic (at 146 ft AGE) before 
conducting a go-around. The EP was aware of the incursion after having received an audible alert 
(BAMO display concept) but continued to descend to visually acquire the traffic to confirm the alert. 
This incident would have been classified as a Category D incursion if the EP had initiated the go-around 
at first awareness of the alert. Post-experimental briefings confirmed that the pilot was aware of the 
traffic but, because of the information provided by the display concept, felt safe proceeding lower in 
altitude to visually confirm the hazard. 

No statistically significant differences were found between the display conditions for both the 
distance from ownship to the incurring traffic or for the difference in time (i.e.reaction time) from the 
incursion event to when a go around was initiated by the pilots, based on a Multivariate Analysis of 
Variance (MANOVA) (p > 0.05); therefore, separate ANOVAs were not conducted on these dependent 
variables. 


Table 4. Algorithm Performance for Rare Event Scenario. 



Distance to Traffic 

Time to Traffic 

Mean (ft) 

Std. Dev. (ft) 

Mean (see) 

Std. Dev. (see.) 

PathProx^M Caution 

5836 

348.5 

35 

2.0 

PathProx^M Warning 

4234 

268.0 

25 

2.0 

RSM Warning 

4060 

135.7 

24 

2.6 
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Data on both incursion detection algorithms were collected during the incursion events; however, 
only the alerts generated by the RSM algorithm were displayed to the EP, when the display condition 
included alerting. The alerting algorithm performance for the rare event incursion scenarios trials are 
presented in Table 4. Data from two EPs were omitted as outliers from these means because the pilots 
were initially heading to the wrong runway and turned toward Runway 34R within 1.1 nm. Even though 
alerts were generated the results were skewed. The “Time to Traffic” in Table 4 is estimated based on the 
relative distance from the traffic and the relative ground speed of the ownship at the time of the alert. A 
MANOVA showed a significant effect between the incursion detection algorithms tested (F(4,42) = 
190.163). Subsequent Analysis of Variances (ANOVAs) on the dependent variables revealed that the 
PathProx^'^ Caution alert was generated significantly earlier (F(2,42) = 6.839) and at a greater distance 
from the incursion aircraft (F(2,42) = 7.302) than either the PathProx^'^ Warning alert or the RSM 
Warning alert. There were no significant differences between the PathProx^'^ Warning alert and RSM 
Warning alert. 

Of the 16 trials, 

• 8 EPs initially became aware of the incursion traffic by viewing out the window, before the 
alert would have occurred; 

• 5 visually acquired the traffic out the window after an incursion alert would have occurred if 
it were active; 

• 1 saw the traffic on the surface map well before the alert occurred; and 

• 2 EPs did not see the traffic at all (i.e., the Category A and B incursions described above). 

The number of EP’s for each initial traffic awareness location, categorized by display condition, are 

shown in Table 5. 


Table 5. Initial Traffic Awareness Location during Rare Event Scenarios (16 pilots). 


Display Condition 

Out the Window, Before 
Alert Threshold 

Out the Window, 
After Alert Threshold 

Surface Map, Before 
Alert Threshold 

Did Not See 
Traffic 

BMO 

1 

3 

0 

0 

BMOT 

2 

1 

0 

1 

BAMO 

3 

0 

0 

1 

BAMOT 

2 

1 

1 

0 

Total 

8 

5 

1 

2 


As shown in Table 6, the EP’s initial awareness of the traffic was not affected by display concept 
(i.e., differences were not statistically significant at the a = 0.05 level); however, the data indicates a trend 
(p = 0.09) where the incursion traffic was acquired sooner when the EP was provided with traffic on the 
surface map and/or incursion alerts. The data for the two EPs that did not acquire the traffic (BMOT and 
BAMO display conditions) and the two EPs that headed to the incorrect runway (BMO and BAMOT 
display conditions) were omitted from Table 6. 


Table 6. Initial Traffic Awareness Measurements during Rare Event Scenarios (12 pilots). 



Distance to Traffic 

Time to Traffic 

Display Condition 

Mean (ft) 

Std. Dev. (ft) 

Mean (see) 

Std. Dev. (sec.) 

BMO 

3927 

575.6 

24 

1.7 

BMOT 

4910 

1171.3 

29 

7.6 

BAMO 

5261 

622.3 

28 

7.1 

BAMOT 

4745 

1598.1 

28 

9.5 
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For those displays that had alerting (BAMO, BAMOT), there were no significant differences in 
qualitative measures of timeliness of the alerting in terms of being able to take evasive action. Flowever, 
when pilots were asked to rate all four display concepts on the perceived efficacy of the alerts (F(3,16) = 
10.948) and the additional safety value added (F(3,16) = 8.814), an ANOVA revealed a significant effect 
between the displays (p < 0.05). Subsequent post-hoc Student Newman Kuels tests showed that pilots 
reported that the BMO display condition was significantly lower in perceived efficacy and safety value 
added than the other three display conditions. Comparisons between the remaining three display 
conditions did not yield any significant statistical differences in qualitative ratings for these dependent 
variables 


Usability Study Phase 

All test trials conducted during the usability study phase included incursion events. Although the EPs 
were aware that each trial would contain an incursion event, they were not told the type of incursion 
before the initial trial for each scenario. During each test trial, data were simultaneously collected on the 
performance of both incursion detection algorithms; however, only one method, when dictated by the 
experimental configuration, was chosen for flight deck alerting. 

The EPs evaluated each incursion scenario with the seven display conditions to determine the 
effectiveness of the display for runway incursion prevention. The initial trial evaluated the baseline (B) 
display condition. This was done to determine if the EP would visually acquire the incursion traffic out 
the window since the surface map and alerting were not available. The RSM algorithm was used as the 
alert source for display in these trials. The EPs then evaluated both the RSM and PathProx^'^ incursion 
detection algorithms using the five incursion scenarios and the BRIPS display condition. 

During the course of usability data collection, it became apparent that the BMO and BAMO display 
conditions provided nearly identical traffic awareness for the EPs as the B (no alerts) and BA (audible 
alerts only) display conditions, respectively; therefore, a limited number of BMO and BAMO test trials 
were conducted in the interest of time. 

Quantitative Results 

The alerting algorithm performance is summarized in Table 7. A total of 602 test trials were 
completed. Data were not analyzed for 14 trials due to missing data files and one test trial was omitted 
due to unorthodox maneuvering by the EP, yielding a total of 587 trials. 

Alerts were required for display on 415 test trials. During these 415 trials, the RSM was the alert 
source 81 percent of the time (336 trials). 

RSM generated alerts on 524 of the 587 total trials. RSM did not alert on 60 trials due to the 
maneuver taken by the EP. For example, the EP may have acquired the traffic out the window and 
conducted a go-around before the alerting criteria were met. RSM did not alert on three trials due to the 
scenario timing (the EP maneuvered such that no incursion event occurred). 

Of the possible 587 test trials, PathProx^'^ generated warning alerts on 345 trials. The 242 trials in 
which PathProx™ did not alert were generally due to the maneuvering performed by the EP. For 
instance, when RSM alerting was provided, the pilot executed an avoidance maneuver and this generally 
occurred before the PathProx^'^ alerts were generated. Caution alerts were only possible on 232 trials 
(scenarios 1 and 5) and of these, alerts were generated on 154 trials. Detailed PathProx^'^ analysis was 
not possible because specific alerting criteria are proprietary. 
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Table 7. Summary of Alerts Generated. 


Scenario 

Number of 
Trials 

RSM 

PathProx Caution 

PathProx Warning 

Missing 

File 

Alerts 

No Alert 

Alerts 

No Alert 

Alerts 

No Alert 

1 

117 

97 

20 

116 

1 

98 

19 

4 

2 

117 

113 

4 

N/A 

N/A 

84 

33 

5 

3 

121 

116 

5 

N/A 

N/A 

61 

60 

1 

4 

117 

109 

8 

N/A 

N/A 

64 

53 

2 

5 

115 

89 

26 

38 

77 

38 

77 

3 

Total 

587 

524 

63 

154 

78 

345 

242 

15 


Scenario 1 arrival/takeoff hold results - For the Baseline display (B) condition, all 1 6 EPs acquired 
the incursion traffic out the window for Scenario 1 when approximately 1 nm from the runway threshold. 
As a result, the average go-around initiation point was 443 ft AGE and 5240 ft (approximately 28 
seconds) from the traffic. During the Baseline condition cases, data were still being collected on the 
conflict detection algorithms. On average, the EP initiated a go-around after the PathProx^'^ caution alert 
would have occurred (475 ft AGE and 5641 ft - approximately 34 seconds - from the traffic for the 
baseline condition), but before either the RSM (409 ft AGE and 4272 ft - approximately 24 seconds - 
from the traffic for the baseline condition) or PathProx^’^ warning alert (392 ft AGE and 4117 ft - 
approximately 24 seconds - from the traffic for the baseline condition) would have occurred. 

Algorithm performance analyses for all Scenario 1 test trials are shown in Table 8. The PathProx^’^ 
caution alert was generated much earlier than the warning alerts. Both the RSM and PathProx^’^ warning 
alerts were generated at essentially the same moment. It should be noted that during the test trials in 
which alerts were provided, the EP was asked to continue the maneuver until the alert was received for 
evaluation purposes. 


Table 8. Algorithm Performance for Scenario 1. 



AGL 

Distance to Traffic 

Time to Traffic 

Mean (ft) 

Std. Dev. (ft) 

Mean (ft) 

Std. Dev. (ft) 

Mean (sec) 

Std. Dev. (sec) 

PathProx^M Caution 

613 

98.0 

5757 

325.7 

36 

1.2 

PathProx^M Warning 

496 

88.7 

4175 

278.7 

25 

1.0 

RSM Warning 

492 

92.6 

4115 

114.7 

25 

2.1 


Scenario 2 departure/intersection departure results - Six of 16 EPs (37.5 percent) did not acquire the 
incursion traffic visually for Scenario 2, or saw the traffic too late to abort the departure and actually 
over-flew the runway traffic when using the Baseline (B) display condition. 

Data on the abort initiation location (identified when the throttles and\or ground speed were reduced) 
are shown in Table 9. The EPs aborted at a further distance from Taxiway E when provided with those 
display configurations having alerting (BA, BAMO, BAMOT, and BRIPS). The EPs also aborted the 
departure for all of the trials with alerting provided. Based on measured data, the departure was aborted 
later when alerting was not provided (B, BMO, and BMOT), although still with enough time to stop prior 
to reaching the traffic. Also, as noted above, the EP actually took off on six trials and over-flew the 
traffic, using the Baseline display. 

For this departure scenario, the RSM warning alert was generated before and at a lower ground speed 
than the PathProx^’^ warning alert (Table 10); however, both provided ample time to abort and stop 
before reaching the incurring traffic. PathProx^'^ caution alerts are not generated on departure. 
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Table 9. Abort Initiation during Scenario 2 for Display Conditions. 



Nunber of Abort 
Occurrences 

Distance to Taxiway L 

Time to Taxiway L 

Mean (ft) 

Std. Dev. (ft) 

Mean (sec) 

Std. Dev. (sec.) 

B 

8 

2177 

485.6 

18 

1.9 

BMO 

3 

2004 

595.3 

15 

5.7 

BMOT 

12 

1849 

408.6 

13 

3.4 

BA 

14 

2493 

108.1 

19 

1.1 

BAMO 

5 

2530 

56.5 

19 

1.1 

BAMOT 

15 

2498 

160.0 

19 

1.3 

BRIPS 

16 

2542 

86.0 

19 

0.9 


Table 10. Algorithm Performance for Scenario 2. 



Ground Speed 

Distance from Twy L 

Time from Twy L 

Mean (kt) 

Std. Dev. 
(kt) 

Mean (ft) 

Std. Dev. 
(ft) 

Mean (sec) 

Std. Dev. 
(sec) 

PathProx^M Warning 

41 

4.4 

2452 

75.2 

19 

1.5 

RSM Warning 

28 

4.7 

2637 

102.9 

21 

1.4 


Scenario 3 arrival/departure results - For the Baseline (B) condition, 13 of 16 EPs (81 percent) did 
not acquire the incursion traffic visually for Scenario 3. Due to the scenario design; however, 14 EPs 
landed and stopped before reaching the crossing runway. One EP landed but taxied through the 
intersection as the traffic was departing from crossing Runway 25. One EP conducted a go-around. 

The algorithm performance for Scenario 3 is presented in Table 11. On average, the RSM Warning 
alert was generated slightly before the PathProx^’^ Warning alert. Both algorithms alerted at or slightly 
before touchdown. PathProx^'^ Caution alerts were not generated for this scenario. 

For all Scenario 3 trials (122 total), a go-around was conducted just before touchdown (7 ft AGE) on 
1 1 percent (14) of the trials. 


Table 11. Algorithm Performance for Scenario 3. 



Distance to Runway 34R/25 Intersection 

Time to Runway 34R/25 Intersection 

Mean (ft) 

Std. Dev. (ft) 

Mean (sec) 

Std. Dev. (sec) 

PathProx^M Warning 

1190 

306.4 

9 

2.8 

RSM Warning 

1301 

944.2 

11 

1.4 


Scenario 4 departure/departure results - Five of 16 EPs (31.3 percent) saw the incursion traffic 
visually and aborted the departure when using the Baseline (B) display condition. Nine of 16 EPs (62.5 
percent) did not see the incursion traffic visually, or saw the traffic too late to abort the departure. For 
these trials, the ownship came within an average distance of 458 ft (range from 179 ft to 795 ft) from the 
incursion traffic. Two of the trials resulted in no incursion event due to the scenario timing; therefore, 
these data are not included in the analysis. 

Generally, when alerts were provided (BA, BAMO, BAMOT, and BRIPS), the EPs were aware of the 
traffic conflict and initiated aborts earlier than without the alerts as shown in Table 12. For trials in which 
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the EP visually acquired traffic - either out-the-window or using a map with traffic display - the departure 
was still aborted with enough time to stop before the crossing runway. 

The RSM warning alerts were generated before the PathProx^'^ warning alerts for this departure 
scenario. When RSM alerts were presented, the EP typically reacted based on those alerts; therefore, 
PathProx^'^ alerts were sometimes not generated or generated after initiation of the abort maneuver. For 
an accurate assessment of PathProx^'^ performance, only the results from the trials in which the 
PathProx^’^ alerts were shown to the EP are presented (Table 13). Again, the RSM warning alert was 
generated before and at a lower ground speed than the PathProx^'^ warning alert; however, both still 
provided ample time to abort and stop before reaching the incurring traffic. PathProx^’^ caution alerts are 
not generated on departure. 


Table 12 , Abort Initiation during Scenario 4 for Display Conditions. 



Distance to Runway 25 

Time to Runway 25 

Mean (ft) 

Std. Dev. (ft) 

Mean (sec) 

Std. Dev. (sec.) 

B 

1194 

799.7 

9 

6.2 

BMO 

1928 

293.4 

15 

2.7 

BMOT 

1743 

531.6 

13 

4.6 

BA 

2091 

65.8 

17 

0.6 

BAMO 

2056 

151.6 

16 

2.1 

BAMOT 

2107 

69.4 

17 

0.9 

BRIPS 

2096 

79.6 

17 

1.2 


Table 13. Algorithm Performance for Scenario 4. 



Ground Speed 

Distance from Rwy 25 

Time from Rwy 25 

Mean (kt) 

Std. Dev. 
(kt) 

Mean (ft) 

Std. Dev. 

(ft) 

Mean (sec) 

Std. Dev. 
(sec) 

PathProx^M Warning 

48 

8.5 

1891 

115.2 

15 

1.8 

RSM Warning 

29 

4.8 

2216 

35.3 

18 

0.9 


Scenario 5 taxi crossing/departure results - Fourteen of 16 EPs (87.5 percent) acquired the incursion 
traffic out the window when using the Baseline (B) condition and stopped before reaching Runway 25. 
One EP did not see the traffic and actually crossed Runway 25 in front of the departing traffic. One EP 
saw the departing traffic out the window but chose to cross Runway 25 anyway. 

The RSM algorithm uses predictive alerting, triggered on the aircraft ground speed, in an attempt to 
keep the aircraft clear of a runway, behind the hold line. If the ownship is traveling 8 kt or greater and is 
not slowing down, the alert will be generated before the aircraft reaches the hold line, providing sufficient 
distance to stop before crossing the hold line. As the taxi speed increases, the alert is generated when the 
ownship is a farther distance from the hold line. However, the alert is not generated until after the 
ownship crosses the hold line when the ownship is traveling less than 8 kt. The 8 kt threshold was used to 
minimize false or nuisance alerts as the ownship taxis toward a hold line. Details on the PathProx^’^ 
implementation are proprietary. 

The effect of predictive alerting with the RSM alert using an 8 kt ground speed trigger is shown in 
Table 14. If the ground speed was above 8 kt approaching the hold line, the RSM alert was generated 
before reaching the hold line (positive value). A negative value indicates the aircraft crossed over the 
hold line before the alert was generated. Note that the hold line was 170 ft from the edge of the runway 
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for this scenario so in all cases the alert was still generated well before the aircraft entered the runway. 
The proactive RSM alerting trigger speed may need to be reassessed to keep a higher percentage of taxi 
blunders from crossing hold lines. These data indicate that almost 40% of the time ownship crossed the 
hold line (i.e., technically, a runway incursion). 

As with Scenario 4, the RSM alerts were generated before the PathProx™ alerts. Therefore, for an 
accurate assessment of PathProx^'^, only the results from trials in which the PathProx™ alerts were given 
to the EP are presented (Table 15). The data shows that PathProx does not use predictive alerting. All 
warning and cautions occurred after the ownship crossed the hold line, but were still triggered 153 and 
130 ft, respectively, prior to the runway. Assuming a reasonable taxi speed, ownship should have been 
able to react and stop prior to the runway. 


Table 14, Scenario 5 RSM Alert Generation 


EP 

Ground 
Speed (kt) 

Distanee to hold 
line (feet) 

1 

9.7 

59 

2 

9.7 

48 

3 

9.2 

59 

4 

7.9 

-12 

5 

7.0 

-6 

6 

8.4 

56 

7 

6.0 

-7 

8 

13.4 

17 

9 

10.3 

70 

10 

8.4 

48 

11 

8.3 

48 

12 

7.5 

-17 

13 

7.4 

-14 

15 

6.5 

-7 

16 

10.6 

8 


Table 15. Algorithm Performance for Scenario 5. 



Distance Before/Past Hold Line 

Distance from Runway 25 Edge 

Mean (ft) 

Std. Dev. (ft) 

Mean (ft) 

Std. Dev. (ft) 

PathProx^M Caution 

17 past 

3.8 

153 

3.8 

PathProx^M Warning 

41 past 

19.5 

130 

19.5 

RSM Warning 

23 before 

32.6 

193 

32.6 
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Qualitative Results 

Post-run (Appendices E and F), post-block (Appendices G and H), and post-test (Appendices I and J) 
questionnaires were administered. These data are discussed in the following. Ratings for most of the 
questions were generally based on a scale of 1 (low) to 10 (high). The result values shown in this section 
are the mean values based on the 1 to 1 0 scale, unless stated otherwise. 

Moving Map Display - Statistical analyses were not conducted on the qualitative data on pilot ratings 
of the moving map display because of limited power, due to the low number of observations, for the non- 
parametric test; therefore, descriptive data are reported below. When the EPs rated the effectiveness of 
the perspective surface map relative to the plan view map for prevention of runway incursions, the 
perspective map was rated slightly more effective (2.88 on a 10-point scale). However, the location of the 
surface map was deemed suboptimal and should be positioned closer to the pilot’s head-up field of view. 
The traffic presentation on the surface map was considered easily discernable (8.38). Most EPs (14 of 16) 
considered traffic presentation necessary to prevent runway incursions. The addition of traffic was rated 
to provide increased (8.56) SA over a surface map with only ownship location. Over half of the EPs (10 
of 16) indicated visual presentation of alerts on the surface map was necessary to prevent runway 
incursions. The addition of visual alerts was rated to provide increased (6.27) SA. The EPs considered 
the terms used for the incursion alerts (e.g. “Warning, Traffic Departing 25”) to be very effective (8.88). 

For display conditions with alerting available, the EPs indicated that the incursion event would most 
likely be brought to their attention first through audible alerting, then on the surface map, and lastly out 
the window. Nine of 16 EPs indicated that an audible alert alone would be the minimum necessary for an 
effective incursion prevention display, while five of 1 6 indicated a surface map with ownship and traffic 
but without alerting would be the minimum to be effective. However, all 16 EPs indicated a surface map 
with ownship and traffic in conjunction with an audible alert would be an optimal incursion prevention 
display. According to averaged EP ratings, for all alerting display conditions, the caution and warning 
alerting system was perceived to provide the greatest amount of runway incursion awareness (see Table 
16). Based on the descriptive results, there was no consistency across the EPs in terms of alerting 
preference for incursion awareness and perceived SA improvement. For each of the four display 
conditions (BA, BAMO, BAMOT, BRIPS), the majority of EPs rated the “caution and warning” alerting 
higher for incursion awareness. However, for each display condition, at least 6 of 16 EPs rated the 
“warning only” alerting as preferable for incursion awareness and, for these EPs, the average SA rating 
was 6.5 to 7.7 in perceived SA improvement. 

Table 16. Alerting Preference for Incursion Awareness and Perceived SA Improvement 


Alerting Type 

Warning Only 

Caution & Warning 

Number of EPs 

SA 

Rating 

Number of EPs 

SA 

Rating 

BA 

6 

6.5 

10 

4.2 

BAMO 

6 

6.7 

10 

4.3 

BAMOT 

7 

7.7 

9 

5 

BRIPS 

7 

7.6 

9 

5.2 


Note: “Number of EPs” column represents the number of EPs that indicated preference for alerting 
condition. “SA Rating” column represents the amount of SA improvement (0-10 scale) for the alerting 
condition indicated by that EP. 

Algorithm Alerting - All EPs indicated that both the RSM and PathProx^’^ alerting provided 
sufficient time to avoid a potential conflict. Only six of 16 EPs (37.5 percent) thought providing both 
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caution and warning alerts was more effective in preventing runway incidents than a single warning alert. 
However, nine of 16 EPs indicated that the caution and warning system provided greater (4.39) SA and 
provided more reaction time. Eleven of 16 EPs (68.8 percent) liked the idea of having a caution alert in 
conjunction with a warning to provide more evaluation and reaction time, i.e. a greater comfort level. For 
the scenarios evaluated, the EPs generally felt that providing caution and warning alerts on approach was 
most effective, while a warning alert alone was sufficient when on the airport surface (during departure 
and taxi). 

The EPs were asked if resolutions or maneuver guidance should be provided in addition to runway 
incursion alerts for various operations. The majority of EPs would like to be provided with maneuver 
guidance for conflict resolution on final approach (12 of 16) and when taxiing across a runway (nine of 
16). Half of the EPs would like maneuver guidance on departure. 

Surface Operations Safety - In general, the EPs felt safer during runway incursion encounters when 
alerting was provided (B=2.06, BMO=2.75, BMOT= 5.38, BA=7.19, BAMO=7.44, BAMOT=9.31, and 
BRIPS=9.19). Analysis of pilot responses to their perceived safety and runway incursion prevention 
effectiveness support this conclusion with significant main effects found (i.e., for perceived safety 
(F(6,90) = 857.390) and runway incursion prevention effectiveness (F(6, 90) = 188.793)). The addition 
of traffic was marginally beneficial when presented on a moving map display and was only effective 
when alerting was provided. A possible cause may be that pilots ARE out-the-window and CANNOT BE 
focused on a head-down display during this critical phase of flight, especially in VMC flight conditions. 
When alerted, the pilot is provided a cue that it might be beneficial for them to direct focus and attention 
to the head-down display to locate the incurring traffic. In fact, pilots rated having audible alerts (BA) 
and having alerts with a map with ownship but no traffic (BAMO) similarly for runway incursion 
prevention on almost all dependent variables measured. For the experimental scenarios tested, the 
moving map display appeared beneficial in preventing incursions only when both traffic AND alerting 
were included. Ideally, a system should have the ability to alert the pilot to incurring events and cue them 
where to look without having to go head-down, into the cockpit. Head-up and head-worn displays 
provide this type of capability but are to date, not generally installed in low-end GA aircraft. 

Conclusions 

A Runway Incursion Prevention System (RIPS), developed for commercial transport flight decks and 
adapted for general aviation operations, was evaluated in a piloted simulation study. The purpose of the 
study was to evaluate the aircraft-based incursion detection algorithms and associated alerting and airport 
surface display concepts for general aviation operations using a rare event study followed by a usability 
study. 

The results indicate that, during the rare event testing, for the scenario evaluated, most pilots were 
able to acquire the incurring traffic by looking out the cockpit windows (in VMC conditions), even before 
incursion alerting was activated. In the usability study, for some scenarios, pilots were generally unable 
to detect the incurring traffic out-the-window. This emphasized the importance of providing alerts of 
potential runway conflicts to the pilot. 

The surface map showing ownship (without other traffic) was rated as being significantly inferior to a 
surface map showing traffic and/or incursion alerting for perceived safety value added. However, one 
pilot experienced a severe runway incursion and risk for collision despite having traffic displayed on a 
surface map. The addition of traffic was marginally beneficial when presented on a surface map display 
and was only effective when alerting was provided. A possible cause may be that pilots had transitioned 
heads-up out the window and were not focused on the head-down display to locate the incurring traffic. 
Pilots reported that the utility of the surface map would be significantly more effective if located higher 
on the instrument panel closer to the pilot’s head-up field of view. 

The results generally match past research on commercial operations in that the incursion alerts 
provided sufficient time to avoid a potential incursion conflict. Departures were generally aborted sooner 
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when alerts were provided, resulting in greater safety margins. A surface map showing ownship and 
traffic along with audible alerts was considered an optimal incursion prevention display, while an audible 
alert alone was considered a minimally effective display. Over half of the pilots would like maneuver 
guidance for conflict resolution in conjunction with incursion alerting. In general, the pilots felt 
substantially safer during potential runway incursion incidents with alerting onboard. 
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Evaluation Pilot 


Appendix A: Rare Event Test Matrix 

EPn (n=l to 16) 

RIPS Scenario Scenario 1 - arrival / takeoff hold (see Figure 8) 

Meteorological VMCl = 3 nm, day, 1000 ft ceiling 

VMC2 = 3 nm, day, 2000 ft ceiling 
IMCl = 1 nm, day, 1000 ft ceiling 
1MC2 = 1 nm, day, 400 ft ceiling 
IMC3 = 2 nm, day, 1 000 ft ceiling 


Altitude (ft) 

4412 

(OAGL) 

4420 

(8AGL) 

5420 

(1008 AGL) 

7100+ 
(2688 AGL) 

Wind Speed (kt) 

10 

10 

20 

30 

Wind Direetion (deg) 

260 

260 

260 

360 


Note: gradual shift of wind speed and direction 


Display Concept BMO = Baseline + surface map and ownship 

BMOT = Baseline + surface map and ownship and traffic 

BAMO = BMO + audible incursion alerts 

BAMOT = BMOT + incursion alerts (audible & visual) 

Note: IC = Initial Condition 

Approach S34RE = Straight- in approach to 34R and land. IC is 3nm out, 1010 ft AGE, 90 kt 
S34RW= Straight-in approach to 34R then waveoff at 200 ft AGE. IC is 3 nm out, 

1010 ft AGE, 90 kt 

S34ESS= Straight-in approach, lined up with 34E, sidestep to 34R and land. IC is 3nm 
out, 1010 ft AGE, 90 kt 

S25E = Straight-in approach to 25 and land. IC is 3 nm out, 1800 ft AGE 
C-long = Non-precision approach - initial approach to 34R from base leg ending in 
Circling approach to 25 and land. IC is 9.56 nm out from 34R on left 
base (85 degree heading & 1.6 nm left offset), 2737.5 ft AGE 
S-long = lES approach - approach to 34R from base leg, follow approach path and go 

around at 200 ft. IC is 9.56 nm from 34R on left base (85 degree heading & 1.6 
nm left offset), 2737.5 ft AGE 


24 



Rare Event Test Matrix 


EP 

Case No. 

RIPS 

Scenario 

Meteoro- 

logical 

Alert 

Source 

Display 

Concept 

Approach 

1,5,9,13 

1 

None 

VMCl 

N/A 

BMO 

S34RL 


2 

Scenario 1 

VMCl 

N/A 

BMO 

S34RL 


3 

None 

IMC2 

N/A 

BMO 

S34RL 


4 

None 

IMC2 

N/A 

BMO 

S34RL 


5 

None 

VMCl 

N/A 

BMO 

S34RW 


6 

None 

VMCl 

N/A 

BMO 

S34RW 


7 

None 

IMC2 

N/A 

BMO 

S34RW 


8 

None 

IMC2 

N/A 

BMO 

S34RW 


9 

None 

VMCl 

N/A 

BMO 

S34LSS 


10 

None 

VMCl 

N/A 

BMO 

S34LSS 


11 

None 

VMCl 

N/A 

BMO 

S34LSS 


12 

None 

VMCl 

N/A 

BMO 

S34LSS 


13 

None 

VMC2 

N/A 

BMO 

S25L 


14 

None 

VMC2 

N/A 

BMO 

S25L 


15 

None 

VMC2 

N/A 

BMO 

S25L 


16 

None 

VMC2 

N/A 

BMO 

S25L 


17 

None 

IMC3 

N/A 

BMO 

C-long 


18 

None 

IMC3 

N/A 

BMO 

C-long 


19 

None 

IMC2 

N/A 

BMO 

S-long 

2,6,10,14 

20 

None 

VMCl 

N/A 

BMOT 

S34RL 


21 

Scenario 1 

VMCl 

N/A 

BMOT 

S34RL 


22 

None 

IMC2 

N/A 

BMOT 

S34RL 


23 

None 

IMC2 

N/A 

BMOT 

S34RL 


24 

None 

VMCl 

N/A 

BMOT 

S34RW 


25 

None 

VMCl 

N/A 

BMOT 

S34RW 


26 

None 

IMC2 

N/A 

BMOT 

S34RW 


27 

None 

IMC2 

N/A 

BMOT 

S34RW 


28 

None 

VMCl 

N/A 

BMOT 

S34LSS 


29 

None 

VMCl 

N/A 

BMOT 

S34LSS 


30 

None 

VMCl 

N/A 

BMOT 

S34LSS 


31 

None 

VMCl 

N/A 

BMOT 

S34LSS 


32 

None 

VMC2 

N/A 

BMOT 

S25L 


33 

None 

VMC2 

N/A 

BMOT 

S25L 


34 

None 

VMC2 

N/A 

BMOT 

S25L 


35 

None 

VMC2 

N/A 

BMOT 

S25L 


36 

None 

IMC3 

N/A 

BMOT 

C-long 


37 

None 

IMC3 

N/A 

BMOT 

C-long 
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38 

None 

IMC2 

N/A 

BMOT 

S-long 

3,7,11,15 

39 

None 

VMCl 

N/A 

BAMO 

S34RL 


40 

Scenario 1 

VMCl 

RSM 

BAMO 

S34RL 


41 

None 

IMC2 

N/A 

BAMO 

S34RL 


42 

None 

IMC2 

N/A 

BAMO 

S34RL 


43 

None 

VMCl 

N/A 

BAMO 

S34RW 


44 

None 

VMCl 

N/A 

BAMO 

S34RW 


45 

None 

IMC2 

N/A 

BAMO 

S34RW 


46 

None 

IMC2 

N/A 

BAMO 

S34RW 


47 

None 

VMCl 

N/A 

BAMO 

S34LSS 


48 

None 

VMCl 

N/A 

BAMO 

S34LSS 


49 

None 

VMCl 

N/A 

BAMO 

S34LSS 


50 

None 

VMCl 

N/A 

BAMO 

S34LSS 


51 

None 

VMC2 

N/A 

BAMO 

S25L 


52 

None 

VMC2 

N/A 

BAMO 

S25L 


53 

None 

VMC2 

N/A 

BAMO 

S25L 


54 

None 

VMC2 

N/A 

BAMO 

S25L 


55 

None 

IMC3 

N/A 

BAMO 

C-long 


56 

None 

IMC3 

N/A 

BAMO 

C-long 


57 

None 

IMC2 

N/A 

BAMO 

S-long 

4,8,12,16 

58 

None 

VMCl 

N/A 

BAMOT 

S34RL 


59 

Scenario 1 

VMCl 

RSM 

BAMOT 

S34RL 


60 

None 

IMC2 

N/A 

BAMOT 

S34RL 


61 

None 

IMC2 

N/A 

BAMOT 

S34RL 


62 

None 

VMCl 

N/A 

BAMOT 

S34RW 


63 

None 

VMCl 

N/A 

BAMOT 

S34RW 


64 

None 

IMC2 

N/A 

BAMOT 

S34RW 


65 

None 

IMC2 

N/A 

BAMOT 

S34RW 


66 

None 

VMCl 

N/A 

BAMOT 

S34LSS 


67 

None 

VMCl 

N/A 

BAMOT 

S34LSS 


68 

None 

VMCl 

N/A 

BAMOT 

S34LSS 


69 

None 

VMCl 

N/A 

BAMOT 

S34LSS 


70 

None 

VMC2 

N/A 

BAMOT 

S25L 


71 

None 

VMC2 

N/A 

BAMOT 

S25L 


72 

None 

VMC2 

N/A 

BAMOT 

S25L 


73 

None 

VMC2 

N/A 

BAMOT 

S25L 


74 

None 

IMC3 

N/A 

BAMOT 

C-long 


75 

None 

IMC3 

N/A 

BAMOT 

C-long 


76 

None 

IMC2 

N/A 

BAMOT 

S-long 
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Appendix B: Usability Study Test Matrix 

RIPS Scenario 1 = Arrival / takeoff hold 

2 = Departure / intersection departure 

3 = Arrival / departure 

4 = Departure / departure 

5 = Taxi crossing / departure 


Display Concept 


B = Baseline round dials 

BMO = B + surface map and ownship 

BMOT = B + surface map and ownship and traffic 

BA = B + audible incursion alerts 

BAMO = BMO + audible incursion alerts 


BAMOT = BMOT + incursion alerts (audible & visual) 
BRIPS = B + perspective surface map 


All runs conducted under VMCl (3nm, day, 1000 ft ceiling, no wind) 


Table B.l: Display Concept Evaluation for Runway Incursion Prevention 


Case No. RIPS Alert 

Scenario Source 

Display 

Concept 

77 

1 

N/A 

B 

78 

1 

N/A 

BMO 

79 

1 

N/A 

BMOT 

80 

1 

RSM 

BA 

81 

1 

RSM 

BAMO 

82 

1 

RSM 

BAMOT 

83 

1 

RSM 

BRIPS 

84 

2 

N/A 

B 

85 

2 

N/A 

BMO 

86 

2 

N/A 

BMOT 

87 

2 

RSM 

BA 

88 

2 

RSM 

BAMO 

89 

2 

RSM 

BAMOT 

90 

2 

RSM 

BRIPS 

91 

3 

N/A 

B 

92 

3 

N/A 

BMO 

93 

3 

N/A 

BMOT 

94 

3 

RSM 

BA 

95 

3 

RSM 

BAMO 

96 

3 

RSM 

BAMOT 

97 

3 

RSM 

BRIPS 

98 

4 

N/A 

B 

99 

4 

N/A 

BMO 




100 

4 

N/A 

BMOT 

101 

4 

RSM 

BA 

102 

4 

RSM 

BAMO 

103 

4 

RSM 

BAMOT 

104 

4 

RSM 

BRIPS 

105 

5 

N/A 

B 

106 

5 

N/A 

BMO 

107 

5 

N/A 

BMOT 

108 

5 

RSM 

BA 

109 

5 

RSM 

BAMO 

110 

5 

RSM 

BAMOT 

111 

5 

RSM 

BRIPS 


Table B.2: RSM and PathProx^M Incursion Detection Algorithm Evaluation 


Case No. 

RIPS 

Scenario 

Alert Source 

Display 

Concept 

148 

1 

RSM 

BRIPS 

150 

1 

PathProx^M 

BRIPS 

152 

2 

RSM 

BRIPS 

154 

2 

PathProxT^i 

BRIPS 

156 

3 

RSM 

BRIPS 

158 

3 

PathProxT^i 

BRIPS 

160 

4 

RSM 

BRIPS 

162 

4 

PathProx^M 

BRIPS 

164 

5 

RSM 

BRIPS 

166 

5 

PathProx^M 

BRIPS 
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Appendix C: Rare Event Case Lists 

Tables C.l and C.2 lists the experimental case order for each evaluation pilot for the rare event 
portion of the testing. Refer to Appendix A for a detailed description of each case. 


Table C.l. Rare Event Case Order for Evaluation Pilots 1 through 8 



EPl 

EP2 

EP3 

EP4 

EPS 

EP6 

EP7 

EPS 

Run 1 

17 

36 

55 

74 

17 

36 

55 

74 

Run 2 

16 

33 

49 

65 

15 

31 

49 

60 

Run 3 

10 

25 

47 

62 

11 

24 

51 

64 

Run 4 

6 

30 

48 

61 

7 

33 

44 

68 

Run 5 

11 

23 

54 

73 

4 

28 

48 

71 

Run 6 

13 

35 

46 

71 

14 

23 

41 

62 

Run? 

7 

34 

43 

64 

13 

32 

50 

66 

Runs 1 

8 1 

27 1 

45 

1 1 

1 

1 22 1 

39 1 

58 

Run 9 

12 

26 

42 

67 

10 

35 

47 

61 

Run 10 

15 

29 

52 

60 

5 

25 

45 

70 

Run 1 1 1 

19 

38 

57 

76 1 

19 

38 j 

57 

76 

Run 12 

3 

31 

44 

66 

16 

26 

46 

73 

Run 13 

1 

22 

51 

70 

12 

20 

54 

65 

Run 14 1 

9 

24 

53 

72 1 

6 

29 j 

43 

72 

Run 15 

14 

28 

39 

68 

8 

34 

52 

69 

Run 16 

4 

20 

41 

58 

3 

27 

42 

63 

Run 17 1 

5 

32 

50 

63 1 

9 

30 j 

53 

67 

Run 18 

18 

37 

56 

75 

18 

37 

56 

75 

Run 19 

2 

21 

40 

59 

2 

21 

40 

59 




Table C.2. Rare Event Case List for Evaluation Pilots 9 through 16 



EP9 

EPIO 

EPll 

EP12 

EP13 

EP14 

EP15 

EP16 

Run 1 

17 

36 

55 

74 

17 

36 

55 

74 

Run 2 

3 

28 

44 

68 

12 

34 

54 

67 

Run 3 

11 

32 

53 

71 

8 

27 

47 

60 

Run 4 

1 

35 

39 

69 

3 

23 

41 

63 

Run 5 

13 

23 

42 

72 

1 

25 

46 

64 

Run 6 

16 

31 

47 

58 

14 

33 

52 

70 

Run? 

8 

33 

49 

62 

5 

32 

43 

68 

Runs 

12 

30 

50 

63 

4 

22 

42 

69 

Run 9 

14 

29 

51 

65 

6 

35 

39 

61 

Run 10 

9 

34 

48 

73 

13 

29 

50 

65 

Run 11 

19 

38 

41 

76 

19 

38 

57 

76 

Run 12 

6 

20 

57 

66 

15 

31 

49 

58 

Run 13 

10 

26 

54 

61 

7 

28 

44 

71 

Run 14 

15 

24 

43 

60 

9 

24 

51 

73 

Run 15 

5 

27 

45 

64 

16 

20 

45 

66 

Run 16. 

7 

22 

52 

67 

10 

26 

48 

62 

Run 17 ■ 

4 

25 

46 

70 

11 

30 

53 

72 

Run 18 ‘ ‘ 

18 

37 

56 

75 

18 

37 

56 

75 

Run 19 

2 

21 

40 

59 

2 

21 

40 

59 
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Appendix D: Usability Study Case Lists 

Tables D. 1 through D.4 lists the experimental case order for each evaluation pilot for the usability 
study portion of the testing. Refer to Appendix B for a detailed description of each case. In Tables D1 
and D2, the cases shown with ‘skip’ were not conducted. 

Parti: Display Concept Evaluation 


Table D.l. Case List for Display Concept Evaluation for Pilots 1 through 8 


EPl 

EP2 

EP3 

EP4 

EPS 

EP6 

EP7 

EPS 

77 

77 

84 

84 

91 

91 

98 

98 

78 

83 

90 

85 

92 

97 

104 

99 

79 

82 

89 

86 

93 

96 

103 

100 

80 

81 

88 

87 

94 

95 

102 

101 

81 

80 

87 

88 

95 

94 

101 

102 

82 

79 

86 

89 

96 

93 

100 

103 

83 

78 

85 

90 

97 

92 

99 

105 

84 

84 

91 

91 

98 

98 

105 

105 

90 

85-skip 

92-skip 

97 

104 

99-skip 

106-skip 

111 

89 

86 

93 

96 

103 

100 

107 

no 

88 

87 

94 

95-skip 

102 

101 

108 

109-skip 

87 

88-skip 

95-skip 

94 

101 

102 -skip 

109-skip 

108 

86 

89 

96 

93 

100 

103 

no 

107 

85 

90 

97 

92-skip 

99-skip 

104 

111 

106-skip 

91 

91 

98 

98 

105 

105 

77 

77 

92 

97 

104 

99-skip 

106-skip 

111 

83 

78-skip 

93 

96 

103 

100 

107 

no 

82 

79 

94 

95-skip 

102-skip 

101 

108 

109-skip 

81 -skip 

80 

95 

94 

101 

102 -skip 

109-skip 

108 

80 

81 -skip 

96 

93 

100 

103 

no 

107 

79 

82 

97 

92-skip 

99-skip 

104 

111 

106-skip 

78-skip 

83 

98 

98 

105 

105 

77 

77 

84 

84 

104 

99-skip 

106-skip 

111 

83 

78-skip 

85 -skip 

90 

103 

100 

107 

no 

82 

79 

86 

89 

102 

101 

108 

109-skip 

81 -skip 

80 

87 

88-skip 

101 

102 -skip 

109-skip 

108 

80 

81 -skip 

88-skip 

87 

100 

103 

no 

107 

79 

82 

89 

86 

99 

104 

111 

106-skip 

78-skip 

83 

90 

85-skip 

105 

105 

77 

77 

84 

84 

91 

91 

106 

111 

83 

78-skip 

85-skip 

90 

97 

92-skip 

107 

no 

82 

79 

86 

89 

96 

93 

108 

109-skip 

81 -skip 

80 

87 

88-skip 

95-skip 

94 

109 

108 

80 

81 -skip 

88-skip 

87 

94 

95-skip 

110 

107 

79 

82 

89 

86 

93 

96 

111 

106-skip 

78-skip 

83 

90 

85-skip 

92-skip 

97 
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Table D.2. Case List for Display Concept Evaluation for Pilots 9 through 16 


EP9 

EPIO 

EPll 

EP12 

EP13 

EP14 

EP15 

EP16 

105 

105 

77 

77 

84 

84 

91 

91 

106 

111 

78 

83 

85 

90 

92 

97 

107 

no 

79 

82 

86 

89 

93 

96 

108 

109 

80 

81 

87 

88 

94 

95 

109 

108 

81 

80 

88 

87 

95 

94 

110 

107 

82 

79 

89 

86 

96 

93 

111 

106 

83 

78 

90 

85 

97 

92 

77 

77 

91 

91 

98 

98 

105 

105 

83 

78-skip 

97 

92-skip 

104 

99-skip 

111 

106-skip 

82 

79 

96 

93 

103 

100 

no 

107 

81 

80 

95-skip 

94 

102-skip 

101 

109-skip 

108 

80 

81 -skip 

94 

95-skip 

101 

102 -skip 

108 

109-skip 

79 

82 

93 

96 

100 

103 

107 

no 

78-skip 

83 

92-skip 

97 

99-skip 

104 

106-skip 

111 

84 

84 

105 

105 

77 

77 

84 

84 

85-skip 

90 

106-skip 

111 

78-skip 

83 

85 -skip 

90 

86 

89 

107 

no 

79 

82 

86 

89 

87 

88-skip 

108 

109-skip 

80 

81 -skip 

87 

88-skip 

88-skip 

87 

109-skip 

108 

81 -skip 

80 

88-skip 

87 

89 

86 

no 

107 

82 

79 

89 

86 

90 

85-skip 

111 

106-skip 

83 

78-skip 

90 

85-skip 

91 

91 

84 

84 

91 

91 

98 

98 

97 

92-skip 

90 

85-skip 

97 

92-skip 

104 

99-skip 

96 

93 

89 

86 

96 

93 

103 

100 

95-skip 

94 

88-skip 

87 

95-skip 

94 

102-skip 

101 

94 

95-skip 

87 

88-skip 

94 

95-skip 

101 

102 -skip 

93 

96 

86 

89 

93 

96 

100 

103 

92-skip 

97 

85 -skip 

90 

92-skip 

97 

99-skip 

104 

99 

98 

98 

98 

105 

105 

77 

77 

99-skip 

104 

99-skip 

104 

106-skip 

111 

78-skip 

83 

100 

103 

100 

103 

107 

no 

79 

82 

101 

102 -skip 

101 

102 -skip 

108 

109-skip 

80 

81 -skip 

102 -skip 

101 

102 -skip 

101 

109-skip 

108 

81 -skip 

80 

103 

100 

103 

100 

no 

107 

82 

79 

104 

99-skip 

104 

99-skip 

111 

106-skip 

83 

78-skip 
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Part 2: Incursion Detection Algorithm Evaluation 


Table D.3. Case List for Incursion Detection Algorithm Evaluation for Pilots I through 8 


EPl 

EP2 

EP3 

EP4 1 

EPS 

EP6 

EP7 

EPS 

148 

150 

148 

150 

152 

154 

152 

154 

150 

148 

150 

148 

154 

152 

154 

152 

154 

152 

154 

152 

162 

160 

162 

160 

152 

154 

152 

154 

160 

162 

160 

162 

156 

158 

156 

158 

148 

150 

148 

150 

158 

156 

158 

156 

150 

148 

150 

148 

162 

160 

162 

160 

158 

156 

158 

156 

160 

162 

160 

162 

156 

158 

156 

158 

164 

166 

164 

166 

164 

166 

164 

166 

166 

164 

166 

164 

166 

164 

166 

164 


Table D.4. Case List for Incursion Detection Algorithm Evaluation for Pilots 9 through 16 


EP9 

EPIO 

EPll 

EP12 

EP13 

EP14 

EP15 

EP16 

164 

166 

164 

166 

148 

150 

148 

150 

166 

164 

166 

164 

150 

148 

150 

148 

162 

1 160 

162 

160 

158 

156 

158 

156 

160 

162 

160 

162 

156 

158 

156 

158 

156 

158 

156 

158 

164 

166 

164 

166 

158 

156 

158 

156 

166 

164 

166 

164 

154 

152 

154 

152 , 

154 

152 

154 

152 

152 

154 

152 

154 

152 

154 

152 

154 

148 

150 

148 

150 

160 

1 162 

160 

162 

150 

148 

150 

148 

162 

160 

162 

160 
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Appendix E: Post Run Questionnaires 

The evaluation pilot completed a questionnaire (see below) at the end of each evaluation run. Mental 
Demand, Physical Demand, Temporal Demand, Performance (Pilot), Effort, and Frustration were all Task 
Load Index (TLX) measures that evaluated workload. Demand on and Supply of Attentional Resources 
and Understanding of Situation values were combined to derive the Situation Awareness Rating 
Technique (SART) rating. Level of Terrain Awareness and Stress were two other independent measures 
that were collected. 


MENTAL DEMAND 


Low 

High 

PHYSICAL DEMAND 
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 

1 1 .1 1 1 

Low 

High 

TEMPORAL DEMAND 

1 1 III 1 1 1 1 1 1 1 1 1 1 

1 1 1 1 1 1 

Low 

High 

PERFORMANCE 

1 1 III 1 1 1 1 1 1 1 1 1 1 

1 1 ll 1 1 

Good 

Poor 


EFFORT 


Low 

High 

FRUSTRATION 
1 1 1 1 1 1 1 1 1 1 1 1 1 

1 1 1 1 1 1 1 1 

Low 

High 


DEMAND ON ATTENTIONAL RESOURCES 


Low High 

SUPPLY OF ATTENTIONAL RESOURCES 


Low High 

UNDERSTANDING OF THE SITUATION 


Low High 

LEVEL OF TERRAIN AWARENESS 


Low High 

STRESS 


Low High 


Workload: 

Situational Awareness: 

TLX - Mental Demand 

SART - Demand on Attentional Resources 

Physical Demand 

Supply of Attentional Resources 

Temporal Demand 

Understanding of the Situation 

Performance 


Effort 


Frustration 
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Run Questionnaire Key: 


Title 

MENTAL DEMAND 


PHYSICAL DEMAND 


Descriptions 

How much mental and perceptual activity 
was required (e.g., thinking, deciding, 
calculating, remembering, looking, 
searching, etc.)? Was the task easy or 
demanding, simple or complex, exacting or 
forgiving? 

How much physical activity was required 
(e.g., pushing, pulling, turning, controlling, 
activating, etc.)? Was the task easy or 
demanding, slow or brisk, slack or 
strenuous, restful or laborious? 


TEMPORAL DEMAND How much time pressure did you feel due to 

the rate or pace at which the tasks or task 
elements occurred? Was the pace slow and 
leisurely or rapid and frantic? 


PERFORMANCE How successful do you think you were in 

accomplishing the goals of the task set by 
the experimenter (or yourself)? How 
satisfied were you with your performance in 
accomplishing these goals? 

EFFORT How hard did you have to work (mentally 

and physically) to accomplish your level of 
performance? 


FRUSTRATION LEVEL How insecure, discouraged, irritated, 

stressed and annoyed versus secure, 
gratified, content, relaxed and complacent 
did you feel during the task? 
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Please rate your overall impression of the scenario in terms of how much attention and effort was required 
to perform the scenario successfully. Things to consider are the degree of instability, complexity, and 
variability that you perceived while flying the scenario. 

DEMAND ON ATTENTIONAL RESOURCES 


Low High 

Please rate your overall impression of the scenario in terms of the amount of “spare” attention to give to 
other tasks. Was 100% of your attention directed towards successfully completing the scenario? Or, 
could you have completed other sub-tasks while flying the scenario? Things to consider include your 
level of arousal, level of concentration, and if you attention was divided across many sub-tasks. 

SUPPLY OF ATTENTIONAL RESOURCES 


Low High 

Please rate your overall understanding of what was happening with the aircraft. Mark on the line below 
the degree to which you felt confident that you were aware of the elements in your environment. Things 
to consider include the level of information quantity and quality as well as familiarity that you felt you 
had with what was taking place during the scenario. 

UNDERSTANDING OF THE SITUATION 


Low High 

Please rate you overall understanding of the terrain environment you were operating within. Things to 
consider for this response are how comfortable were you with your terrain awareness. 

LEVEL OF TERRAIN AWARNESS 


Low High 

Please rate your overall level of stress that you experienced while completing the experimental test run. 

STRESS 


I I I I 1 1 I 


Low 


High 
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1 . Please rate your level of traffic awareness experienced with the display concept during the approach. 

1 = Low; 10 = High 

2 . Please rate your level of awareness of where you were (owns hip position awareness) using the display 
concept during the approach. 1 = Low; 10 = High 

3 . Please rate your ability to stay on path (flight path awareness) using the display concept during the 
approach. 1 = Low; 10 = High 
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Appendix F. Rare Event Run Questionnaire Results 


The following questionnaire was administered to the EP at the conclusion of the rare event runway 
incursion run. Four different display conditions were used for the rare event testing as follows: 


EP# 

Display Condition 

1,5, 9, 13 

BMO 

2, 6, 10, 14 

BMOT 

3, 7, 11, 15 

BAMO 

4, 8, 12, 16 

BAMOT 


Only half of the EPs received incursion alerts for the rare event test run. 


1 . If applicable (BAMO and BAMOT only), please rate the timeliness of the alert (did the alert happen 
in a timely manner) in terms of providing enough time to take evasive action. 

1 = Poor; 10 = Excellent 


EP 

BAMO 

3 

8 

7 

5 

11 

9 

15 

9 

Mean 

7.75 

Std. Dev. 

1.89 


EP 

BAMOT 

4 

7 

8 

8 

12 

10 

16 

9 

Mean 

8.5 

Std. Dev. 

1.29 


2. Upon receiving an incursion alert, was your immediate reaction to : 
Confirm the hazard; Take evasive action; Other; Explain Reason 


EP# 

Confirm Hazard 

Take Evasive Action 

Other 

Reason 

1 

N/A 




2 

N/A 




3 

V 



Would have queried ATC with warning 

4 


V 


Already confirmed traffic prior to alert 

5 

N/A 




6 

N/A 




7 

V 

V 


Need radio response. Initiate each, can 
continue if valid or realign if not. 

8 

V 




9 

N/A 




10 

N/A 




11 


V 
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12 

V 



Unfamiliar with warning 

13 

N/A 




14 

N/A 




15 


V 


Saw first - alert confirmed visual 

16 

V 




Total 

5 

4 




3 . Please indicate where you first perceived the location of the incursion aircraft. 

Window; Surface Map (if applicable); Audible Alert (if applicable) 

If applicable, upon seeing the incursion aircraft, where did you then focus your attention? 

Window; Surface Map (if applicable); Audible Alert (if applicable) 

If applicable, please estimate how useful each of the following were in helping you to detect and avoid 
the runway incursion situation (total 1 00%) 

Window; Surface Map (if applicable); Audible Alert (if applicable) 



First Perceived Location 

Focus Attention 

Detection Usefulness (%) 

EP# 

Window 

Map 

Audible 

Window 

Map 

Audible 

Window 

Map 

Audible 

1 

V 

N/A 

N/A 

V 


N/A 

100 

0 

N/A 

2 

V 


N/A 

V 


N/A 

100 

0 

N/A 

3 


N/A 

V 

V 



10 

0 

90 

4 


V 


V 



40 

60 

0 

5 

V 

N/A 

N/A 

V 


N/A 

100 

0 

N/A 

6 

V 


N/A 

V 


N/A 

80 

20 

N/A 

7 

V 

N/A 


V 



95 

0 

5 

8 

V 




V 


50 

50 

0 

9 

V 

N/A 

N/A 

V 


N/A 

100 

0 

N/A 

10 

- 

- 

N/A 

- 

- 

N/A 

- 

- 

N/A 

11 

V 

N/A 


V 



100 

0 

0 

12 



V 




0 

40 

60 

13 

V 

N/A 

N/A 

V 


N/A 

100 

0 

N/A 

14 

V 


N/A 

- 

- 

N/A 

- 

- 

N/A 

15 

V 

N/A 




V 

80 

0 

20 

16 

V 




V 


33 

33 

34 

Total 

12 

1 

2 

10 

3 

1 

70.6 

14.5 

26.1 
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4, If applicable, how did you use the surface map to detect and locate the incursion aircraft? 


EP 1 - N/A 
EP 2 - N/A 
EP 3 - N/A 

EP 4 - Saw traffic location on the map and then confirmed the traffic by looking out the window. Eooked 
at map afterwards for additional confirmation. 

EP 5 - N/A 
EP 6 - N/A 
EP 7 - N/A 

EP 8 - 1 was looking at the display to confirm my visual out-the-window look when it lit up red. 

EP 9 - N/A 
EP 10 -N/A 
EP 1 1 - N/A 

EP 12 - Upon audible, I confirmed on map then visually. 

EP 1 3 - None 

EP 14 - No response 

EP 15 - Out of field of view. 

EP 16 - Confirm visual. 


If applicable, please indicate whether the surface map alone was sufficient to help you detect and 
locate the intrusion aircraft? Yes; No; Likelihood of detection (%) 


EP# 

Yes 

No 

Likelihood of Deteetion (%) 

1 


V 


2 


V 

0 

3 

N/A 



4 


V 

75 

5 

N/A 



6 

N/A 



7 

N/A 



8 

V 


100 

9 


V 


10 


V 


11 


V 


12 


V 


13 

N/A 



14 


V 


15 


V 


16 


V 


Total 

1 

10 
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What enhancements could be made to the surface map to raise the likelihood? 


EP 2 - More attention getting symbol, location of display is too far from site picture 

EP 4 - Highlight traffic before alert generated. A longer warning on map with audible warning meaning 

imminent danger. 

EP 6 - Eocation 

EP 7 - Traffic of hazard to this aircraft, should have bright/flash indication of hazard with warning 

EP 1 1 - Adding traffic 

EP 12 - Positioning 

EP 14 - Placement 

EP 1 5 - Place within field of view. 

EP 16 - Too cluttered. 


5. If applicable, please indicate whether the audible alert alone was sufficient to help you detect and 
locate the intrusion aircraft? Yes; No; Likelihood (%) 


EP# 

Yes 

No 

Likelihood of Deteetion (%) 

1 

N/A 



2 

N/A 



3 

V 


90 

4 



No answer 

5 

N/A 



6 

N/A 



7 

V 


25 

8 


V 


9 

N/A 



10 

N/A 



11 

V 


100 

12 

V 



13 

N/A 



14 

N/A 



15 

V 



16 


V 


Total 

5 

2 



What enhancements could be made to the audible alert (e.g. 3-D Audio) to raise the likelihood? 

EP 3 - Position 

EP 7 - Need clear beep/squawk to alert about the alert. Plus voice distinct from ATC. 

EP 1 1 - Audio is OK 
EP 12 - None 
EP 15 - Tone 

EP 16 - Female with British accent. 
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6. Please indicate your rating of how much aid the display concept provided in detection of the 
incursion aircraft 1 = None; 10 = Exceptional 


EP 

BMO 

1 

1 

5 

1 

9 

1 

13 

1 

Mean 

1 

Std. Dev. 

0 


EP 

BMOT 

2 

1 

6 

4 

10 

1 

14 

2 

Mean 

2 

Std. Dev. 

1.41 


EP 

BAMO 

3 

8 

7 

2 

11 

8 

15 

7 

Mean 

6.25 

Std. Dev. 

2.87 


EP 

BAMOT 

4 

8 

8 

10 

12 

9 

16 

8 

Mean 

8.75 

Std. Dev. 

0.96 


Collective 


EP 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 


1 

1 

8 

8 

1 

4 

2 

10 

1 

1 

8 

9 

1 

2 

7 

8 

Mean 

4.5 


Std. Dev. 

3.58 


7. Please indicate the likelihood that the display concept would prevent similar runway incursions in 
the real world (based on your experiences). 1 = Low; 10 = High 


EP 

BMO 

1 

1 

5 

1 

9 

5 

13 

1 

Mean 

2 

Std. Dev. 

2 


EP 

BMOT 

2 

10 

6 

9 

10 

8 

14 

5 

Mean 

8 

Std. Dev. 

2.16 


EP 

BAMO 

3 

9 

7 

4 

11 

8 

15 

10 

Mean 

7.75 

Std. Dev. 

2.63 


EP 

BAMOT 

4 

9 

8 

10 

12 

10 

16 

10 

Mean 

9.75 

Std. Dev. 

0.5 


Collective 


EP 

1 

2 


D 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 


1 

10 

9 

9 

1 

9 

4 

10 

5 

8 

8 

10 

1 

5 

10 

10 


Mean 

6.88 

Std. Dev. 

3.5 


8. Please rate your level of perceived safety you believe you would experience if you had this system 
onboard your aircraft during a similar runway incursion event 1 = Low; 10 = High 


EP 

BMO 

1 

1 

5 

1 

9 

5 

13 

5 

Mean 

3 

Std. Dev. 

2.31 


EP 

BMOT 

2 

10 

6 

9 

10 

7 

14 

5 

Mean 

7.75 

Std. Dev. 

2.22 


EP 

BAMO 

3 

7 

7 

7 

11 

6 

15 

10 

Mean 

7.5 

Std. Dev. 

1.73 


EP 

BAMOT 

4 

8 

8 

10 

12 

10 

16 

9 

Mean 

9.25 

Std. Dev. 

0.96 
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Collective 


EP 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 


1 

10 

7 

8 

1 

9 

7 

10 

5 

7 

6 

10 

5 

5 

10 

9 

Mean 

6.88 


Std. Dev. 

2.94 


9. Please rate (if applicable) the effectiveness of the display concept compared to what is currently 
available onboard Part 23 aircraft for prevention of runway incursions. 1 = None; 10 = Exceptional 


EP 

BMO 

1 

1 

5 

1 

9 

1 

13 

5 

Mean 

2 

Std. Dev. 

2 


EP 

BMOT 

2 

10 

6 

10 

10 

7 

14 

5 

Mean 

8 

Std. Dev. 

2.5 


EP 

BAMO 

3 

9 

7 

4 

11 

8 

15 

8 

Mean 

7.25 

Std. Dev. 

2.22 


EP 

BAMOT 

4 

10 

8 

10 

12 

10 

16 

9 

Mean 

9.75 

Std. Dev. 

0.5 


Collective 


EP 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 


1 

10 

9 

10 

1 

10 

4 

10 

1 

7 

8 

10 

5 

5 

8 

9 

Mean 

6.75 


Std. Dev. 

3.45 


Please indicate the reason for your rating. 

EP 1 - The display concept did not show aircraft on it. 

EP 2 - No airport display / map in current aircraft. 

EP 3 - One more piece of information. 

EP 4 - No current indications of traffic available. This display concept gives exceptional traffic 
information. 

EP 5 - System is not indicating traffic. 

EP 6 - Eike having co-pilot and second pair of eyes. 

EP 7 - Definite potential for improvement. 

EP 1 0 - Enhances traffic awareness if used the technology & knew how to use it & part of scan 
EP 1 1 - Eess subjective type of information 

EP 12 - Only visual on Part 23 in low visibility conditions. Audible and surface map make a big 
difference. 

EP 1 3 - No display of traffic. 

EP 15 - Focus attention where many tasks may be involved 

EP 16 - Gives more information than just visual - gets pilot’s attention. 
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10. Please provide any other comments or suggestions for improvement of the efficacy of the display 
concept for prevention of runway incursions. 

EP 1 - Needs to show aircraft on it. 

EP 2 - Ability to use software to make numbers, dials, letters, symbols bigger/smaller, brighter/dimmer. 
EP 6 - Automatic range scaling. 

EP 8 - Because of current vs prior experience. 

EP 1 0 - On instrument approaches would rely more on ATC. 
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Appendix G: Usability Study Display Evaluation Questionnaire Results 


The EPs evaluated the seven display conditions after each test scenario. The acronyms for the display 
conditions are defined as follows: 


B Baseline 

BMO Baseline with Surface Map and Ownship 

BMOT Baseline with Surface Map and Ownship and Traffic 
BA Baseline with Audible Incursion Alerts 

BAMO Baseline with Surface Map and Ownship and Audible Incursion Alerts 

BAMOT Baseline with Surface Map and Ownship, Traffic, and Incursion Alerts 

BRIPS Baseline with Perspective Surface Map 


1 . Please rate the likelihood that the display configuration would prevent the runway incursion 
scenario evaluated. l=low; 10 = high 


Scenario 1: arrival / takeoff hold 


EP# 

B 

BMO 

BMOT 

BA 

BAMO 

BAMOT 

BRIPS 

1 

6 

6 

8 

7 

7 

10 

9 

2 

1 

1 

5 

5 

5 

8 

8 

3 

1 

1 

4 

6 

6 

7 

8 

4 

3 

3 

4 

7 

7 

10 

10 

5 

1 

1 

8 

10 

10 

10 

10 

6 

0 

0 

3 

8 

8 

10 

8 

7 

1 

4 

5 

4 

5 

8 

9 

8 

1 

1 

4 

10 

10 

10 

10 

9 

1 

1 

6 

7 

7 

9 

9 

10 

1 

1 

5 

7 

7 

8 

8 

11 

1 

1 

5 

8 

8 

10 

10 

12 

1 

1 

5 

8 

8 

10 

10 

13 

1 

2 

5 

4 

6 

8 

9 

14 

1 

1 

2 

7 

7 

9 

8 

15 

1 

1 

3 

7 

7 

8 

9 

16 

1 

1 

3 

7 

7 

9 

9 

Mean 

1.38 

1.63 

4.69 

7 

7.19 

9 

9 

Std. Dev. 

1.36 

1.5 

1.66 

1.71 

1.42 

1.03 

0.82 
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Scenario 2: departure / intersection departure 


EP# 

B 

BMO 

BMOT 

BA 

BAMO 

BAMOT 

BRIPS 

1 

6 

6 

8 

7 

7 

10 

10 

2 

1 

2 

6 

6 

6 

8 

9 

3 

1 

1 

3 

6 

6 

8 

8 

4 

2 

2 

7 

7 

7 

10 

10 

5 

1 

1 

3 

9 

9 

9 

10 

6 

0 

0 

3 

8 

8 

10 

8 

7 

1 

3 

4 

4 

5 

9 

8 

8 

1 

1 

4 

10 

10 

10 

10 

9 

1 

3 

6 

7 

8 

9 

9 

10 

1 

1 

5 

6 

6 

9 

9 

11 

1 

1 

6 

8 

8 

10 

10 

12 

1 

1 

5 

10 

10 

10 

10 

13 

1 

2 

5 

4 

7 

9 

8 

14 

1 

2 

2 

7 

7 

9 

8 

15 

1 

1 

2 

7 

7 

7 

8 

16 

1 

2 

3 

7 

7 

9 

9 

Mean 

1.31 

1.81 

4.5 

7.06 

7.38 

9.13 

9 

Std. Dev. 

1.3 

1.38 

1.79 

1.73 

1.41 

0.89 

0.89 


Scenario 3: arrival / departure 


EP# 

B 

BMO 

BMOT 

BA 

BAMO 

BAMOT 

BRIPS 

1 

4 

4 

5 

6 

6 

10 

9 

2 

1 

2 

5 

7 

8 

9 

10 

3 

1 

1 

4 

7 

7 

8 

8 

4 

2 

2 

4 

7 

7 

10 

10 

5 

1 

1 

3 

8 

9 

10 

10 

6 

1 

3 

6 

8 

8 

10 

10 

7 

1 

3 

5 

2 

5 

8 

9 

8 

1 

1 

1 

10 

10 

10 

10 

9 

1 

1 

4 

8 

8 

8 

8 

10 

1 

1 

5 

7 

7 

8 

8 

11 

1 

1 

5 

8 

8 

10 

10 

12 

1 

1 

4 

8 

8 

10 

10 

13 

1 

1 

5 

5 

5 

8 

8 

14 

1 

1 

1 

2 

2 

3 

3 

15 

1 

1 

2 

8 

8 

9 

10 

16 

1 

1 

3 

7 

8 

9 

9 

Mean 

1.25 

1.56 

3.88 

6.75 

7.13 

8.75 

8.88 

Std. Dev. 

0.77 

0.96 

1.5 

2.14 

1.89 

1.77 

1.78 
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Scenario 4: departure / departure 


EP# 

B 

BMO 

BMOT 

BA 

BAMO 

BAMOT 

BRIPS 

1 

4 

4 

6 

6 

6 

10 

10 

2 

1 

2 

6 

8 

8 

9 

10 

3 

1 

1 

4 

7 

7 

8 

8 

4 

1 

2 

5 

6 

6 

9 

9 

5 

0 

1 

3 

10 

10 

10 

10 

6 

0 

2 

6 

8 

8 

10 

8 

7 

1 

4 

5 

5 

6 

9 

9 

8 

1 

1 

3 

10 

10 

10 

10 

9 

1 

1 

4 

8 

8 

8 

8 

10 

1 

1 

6 

7 

8 

8 

8 

11 

1 

1 

7 

8 

8 

10 

10 

12 

1 

1 

4 

10 

10 

10 

10 

13 

1 

3 

5 

5 

7 

9 

8 

14 

1 

1 

3 

7 

7 

9 

8 

15 

1 

1 

3 

7 

7 

8 

8 

16 

1 

1 

3 

7 

7 

9 

9 

Mean 

1.06 

1.69 

4.56 

7.44 

7.69 

9.13 

8.94 

Std. Dev. 

0.85 

1.08 

1.36 

1.59 

1.35 

0.81 

0.93 


Scenario 5: taxi crossins / departure 


EP# 

B 

BMO 

BMOT 

BA 

BAMO 

BAMOT 

BRIPS 

1 

4 

4 

6 

7 

7 

10 

10 

2 

1 

3 

6 

7 

8 

9 

10 

3 

1 

1 

4 

6 

7 

7 

8 

4 

2 

2 

7 

5 

5 

9 

9 

5 

1 

1 

8 

10 

10 

10 

10 

6 

0 

3 

6 

8 

8 

10 

8 

7 

1 

2 

6 

4 

5 

9 

8 

8 

1 

1 

10 

10 

10 

10 

10 

9 

1 

1 

6 

7 

7 

9 

8 

10 

1 

1 

5 

7 

7 

10 

10 

11 

1 

1 

8 

8 

8 

10 

10 

12 

1 

1 

6 

8 

8 

10 

10 

13 

1 

3 

5 

6 

7 

9 

8 

14 

1 

1 

5 

7 

7 

9 

8 

15 

1 

1 

2 

8 

8 

9 

9 

16 

1 

2 

2 

5 

7 

9 

9 

Mean 

1.19 

1.75 

5.75 

7.06 

7.44 

9.31 

9.06 

Std. Dev. 

0.83 

1.0 

2.05 

1.65 

1.36 

0.79 

0.93 
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2 . Please rate the efficacy of the display concept for runway incursion prevention with and without 
alerting for the following display concepts. 1 = low; 10 = high 


Scenario 1 - arrival / takeoff hold 


EP# 

B + no alert 

B + audible alert 

BMO + no alert 

BMO + audible 
alert 

BMO + visual alert 

BMO + audible & 
visual alert 

BMOT + no alert 

BMOT + audible 
alert 

BMOT + visual 
alert 

BMOT + audible & 
visual alert 

BRIPS + no alert 

BRIPS + audible 
alert 

BRIPS + visual 
alert 

BRIPS + audible & 
visual alert 

1 

5 

5 

5 

6 

6 

6 

6 

8 

9 

10 

7 

7 

8 

9 

2 

1 

6 

1 

5 

4 

6 

5 

6 

5 

8 

5 

5 

5 

8 

3 

1 

5 

1 

6 

4 

4 

4 

7 

6 

7 

4 

8 

7 

8 

4 

3 

7 

3 

7 

4 

8 

8 

9 

8 

10 

8 

9 

8 

10 

5 

1 

10 

1 

10 

8 

10 

8 

10 

8 

10 

8 

10 

8 

10 

6 

0 

8 

0 

8 

3 

8 

3 

8 

8 

10 

3 

8 

8 

8 

7 

1 

4 

4 

5 

5 

7 

5 

7 

6 

8 

6 

8 

7 

9 

8 

1 

10 

1 

10 

4 

10 

4 

10 

4 

10 

4 

10 

4 

10 

9 

1 

7 

1 

7 

6 

7 

6 

9 

6 

9 

6 

9 

7 

9 

10 

1 

7 

1 

7 

6 

7 

5 

7 

6 

8 

5 

7 

6 

8 

11 

1 

8 

1 

8 

6 

8 

5 

8 

6 

10 

5 

8 

6 

10 

12 

1 

8 

1 

8 

8 

10 

5 

10 

8 

10 

5 

10 

8 

10 

13 

1 

4 

2 

6 

3 

7 

5 

7 

6 

8 

5 

6 

6 

9 

14 

1 

7 

1 

7 

1 

7 

2 

7 

2 

9 

2 

7 

2 

8 

15 

1 

7 

1 

7 

4 

7 

3 

8 

6 

8 

4 

9 

7 

9 

16 

1 

7 

1 

7 

5 

7 

3 

9 

7 

9 

3 

9 

7 

9 

Mean 

1.31 

6.94 

1.56 

7.13 

4.81 

7.44 

4.81 

8.13 

6.31 

9.0 

5.0 

8.13 

6.5 

9.0 

Std. 

Dev. 

1.14 

1.73 

1.31 

1.45 

1.83 

1.59 

1.68 

1.26 

1.74 

1.03 

1.71 

1.45 

1.67 

0.82 
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Scenario 2 - departure / intersection departure 


EP# 

B + no alert 

B + audible alert 

BMO + no alert 

BMO + audible 
alert 

BMO + visual alert 

BMO + audible & 
visual alert 

BMOT + no alert 

BMOT + audible 
alert 

BMOT + visual 
alert 

BMOT + audible & 
visual alert 

BRIPS + no alert 

BRIPS + audible 
alert 

BRIPS + visual 
alert 

BRIPS + audible & 
visual alert 

1 

6 

7 

6 

7 

7 

7 

8 

9 

9 

10 

8 

9 

9 

10 

2 

1 

6 

2 

6 

5 

5 

6 

6 

5 

8 

6 

8 

7 

9 

3 

1 

6 

1 

6 

4 

7 

3 

6 

4 

8 

4 

6 

4 

8 

4 

2 

7 

2 

7 

4 

8 

7 

9 

8 

10 

7 

9 

8 

10 

5 

1 

10 

1 

10 

8 

10 

7 

10 

8 

10 

8 

10 

9 

10 

6 

0 

8 

0 

8 

3 

8 

3 

8 

6 

10 

3 

8 

6 

8 

7 

1 

4 

3 

5 

4 

6 

4 

8 

7 

9 

5 

7 

6 

8 

8 

1 

10 

1 

10 

1 

10 

4 

10 

4 

10 

4 

10 

4 

10 

9 

1 

7 

3 

8 

4 

8 

6 

8 

6 

9 

5 

9 

6 

9 

10 

1 

6 

1 

6 

5 

6 

5 

6 

5 

9 

5 

6 

5 

9 

11 

1 

8 

1 

8 

7 

9 

6 

10 

9 

10 

7 

10 

9 

10 

12 

1 

10 

1 

10 

8 

10 

1 

10 

8 

10 

1 

10 

8 

10 

13 

1 

4 

2 

7 

5 

7 

5 

7 

7 

9 

4 

7 

7 

8 

14 

1 

7 

2 

7 

2 

7 

2 

7 

2 

9 

2 

7 

2 

8 

15 

1 

7 

1 

7 

2 

6 

2 

7 

3 

7 

3 

7 

3 

8 

16 

1 

7 

2 

7 

5 

8 

3 

9 

7 

9 

3 

9 

7 

9 

Mean 

1.31 

7.13 

1.81 

7.44 

4.63 

7.63 

4.5 

8.13 

6.13 

9.19 

4.69 

8.25 

6.25 

9.0 

Std. 

Dev. 

1.3 

1.82 

1.38 

1.5 

2.09 

1.54 

2.07 

1.5 

2.13 

0.91 

2.09 

1.44 

2.18 

0.89 
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Scenario 3 - arrival / departure 


EP# 

B + no alert 

B + audible alert 

BMO + no alert 

BMO + audible 
alert 

BMO + visual alert 

BMO + audible & 
visual alert 

BMOT + no alert 

BMOT + audible 
alert 

BMOT + visual 
alert 

BMOT + audible & 
visual alert 

BRIPS + no alert 

BRIPS + audible 
alert 

BRIPS + visual 
alert 

BRIPS + audible & 
visual alert 

1 

5 

6 

5 

6 

6 

6 

8 

9 

9 

10 

8 

9 

9 

10 

2 

1 

7 

2 

8 

7 

8 

5 

8 

7 

9 

6 

7 

7 

10 

3 

1 

7 

1 

7 

4 

7 

4 

7 

6 

8 

3 

7 

4 

8 

4 

2 

7 

2 

7 

3 

8 

4 

10 

4 

10 

4 

10 

4 

10 

5 

1 

8 

1 

9 

2 

9 

3 

10 

2 

10 

3 

10 

3 

10 

6 

1 

8 

3 

8 

6 

10 

6 

8 

6 

10 

6 

8 

6 

10 

7 

1 

2 

3 

5 

4 

6 

5 

6 

7 

8 

5 

8 

7 

9 

8 

1 

10 

1 

10 

1 

10 

1 

10 

1 

10 

1 

10 

1 

10 

9 

1 

8 

1 

8 

4 

8 

4 

8 

4 

8 

4 

8 

4 

8 

10 

1 

7 

1 

7 

6 

8 

5 

7 

7 

8 

5 

7 

7 

8 

11 

1 

8 

1 

8 

6 

9 

5 

9 

7 

10 

6 

9 

7 

10 

12 

1 

8 

1 

8 

4 

10 

4 

8 

8 

10 

8 

10 

8 

10 

13 

1 

5 

1 

5 

4 

7 

5 

7 

6 

8 

5 

7 

7 

8 

14 

1 

2 

1 

2 

2 

2 

1 

3 

2 

3 

1 

3 

2 

3 

15 

1 

8 

1 

8 

3 

8 

2 

9 

4 

9 

2 

9 

4 

10 

16 

1 

7 

1 

8 

4 

9 

3 

7 

4 

9 

3 

7 

4 

9 

Mean 

1.31 

6.75 

1.63 

7.13 

4.13 

7.81 

4.06 

7.88 

5.25 

8.75 

4.38 

8.06 

5.25 

8.94 

Std. 

Dev. 

1.01 

2.14 

1.15 

1.89 

1.71 

2.01 

1.81 

1.78 

2.32 

1.77 

2.16 

1.81 

2.29 

1.81 
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Scenario 4 - departure / departure 


EP# 

B + no alert 

B + audible alert 

BMO + no alert 

BMO + audible 
alert 

BMO + visual alert 

BMO + audible & 
visual alert 

BMOT + no alert 

BMOT + audible 
alert 

BMOT + visual 
alert 

BMOT + audible & 
visual alert 

BRIPS + no alert 

BRIPS + audible 
alert 

BRIPS + visual 
alert 

BRIPS + audible & 
visual alert 

1 

5 

6 

5 

6 

6 

6 

8 

9 

9 

10 

8 

9 

9 

10 

2 

1 

8 

2 

8 

7 

8 

6 

8 

8 

9 

7 

8 

7 

10 

3 

1 

7 

1 

7 

6 

8 

4 

7 

6 

8 

4 

8 

5 

8 

4 

1 

6 

2 

6 

5 

7 

5 

7 

5 

9 

5 

7 

5 

9 

5 

1 

10 

1 

10 

3 

10 

3 

10 

3 

10 

3 

10 

3 

10 

6 

0 

8 

2 

8 

2 

8 

6 

8 

6 

10 

2 

8 

6 

10 

7 

1 

5 

4 

6 

5 

7 

5 

7 

6 

9 

5 

7 

6 

9 

8 

1 

10 

1 

10 

5 

10 

3 

10 

5 

10 

2 

10 

2 

10 

9 

1 

8 

1 

8 

4 

8 

4 

8 

4 

8 

4 

8 

4 

8 

10 

1 

7 

1 

8 

6 

8 

6 

8 

6 

8 

6 

8 

6 

8 

11 

1 

8 

1 

8 

8 

10 

7 

10 

10 

10 

7 

10 

10 

10 

12 

1 

10 

1 

10 

4 

10 

4 

10 

4 

10 

4 

10 

4 

10 

13 

1 

5 

3 

7 

5 

7 

5 

7 

6 

9 

3 

6 

5 

8 

14 

1 

7 

1 

7 

2 

7 

3 

7 

3 

9 

2 

7 

2 

8 

15 

1 

7 

1 

7 

3 

7 

3 

7 

4 

8 

3 

7 

4 

8 

16 

1 

7 

1 

7 

5 

7 

3 

7 

5 

9 

5 

9 

7 

9 

Mean 

1.19 

7.44 

1.75 

7.69 

4.75 

8 

4.69 

8.13 

5.63 

9.13 

4.38 

8.25 

5.31 

9.06 

Std. 

Dev. 

1.05 

1.59 

1.24 

1.35 

1.69 

1.32 

1.58 

1.26 

2.00 

0.81 

1.89 

1.29 

2.24 

0.93 
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Scenario 5 - taxi crossins / departure 


EP# 

B + no alert 

B + audible alert 

BMO + no alert 

BMO + audible 
alert 

BMO + visual alert 

BMO + audible & 
visual alert 

BMOT + no alert 

BMOT + audible 
alert 

BMOT + visual 
alert 

BMOT + audible & 
visual alert 

BRIPS + no alert 

BRIPS + audible 
alert 

BRIPS + visual 
alert 

BRIPS + audible & 
visual alert 

1 

4 

6 

4 

6 

5 

6 

6 

8 

7 

10 

6 

8 

7 

10 

2 

1 

7 

3 

8 

7 

8 

6 

8 

7 

9 

6 

8 

8 

10 

3 

1 

6 

1 

5 

4 

7 

4 

7 

6 

7 

5 

7 

6 

8 

4 

2 

5 

2 

5 

5 

6 

7 

9 

7 

9 

6 

9 

6 

9 

5 

1 

10 

1 

10 

9 

10 

8 

10 

9 

10 

8 

10 

9 

10 

6 

0 

8 

3 

8 

6 

8 

6 

8 

6 

10 

8 

8 

6 

8 

7 

1 

4 

2 

5 

4 

6 

6 

8 

7 

9 

5 

7 

6 

8 

8 

1 

10 

1 

10 

5 

10 

10 

10 

10 

10 

10 

10 

10 

10 

9 

1 

7 

1 

7 

5 

7 

6 

9 

6 

9 

6 

8 

6 

8 

10 

1 

7 

1 

7 

4 

8 

5 

7 

6 

10 

5 

7 

6 

10 

11 

1 

8 

1 

8 

9 

9 

8 

10 

9 

10 

8 

10 

9 

10 

12 

1 

8 

1 

8 

6 

8 

6 

10 

9 

10 

6 

10 

9 

10 

13 

1 

6 

3 

7 

6 

8 

5 

8 

8 

9 

6 

7 

7 

8 

14 

1 

7 

1 

7 

2 

7 

5 

8 

5 

9 

5 

8 

4 

8 

15 

1 

8 

1 

8 

7 

8 

2 

8 

7 

9 

2 

8 

7 

9 

16 

1 

5 

2 

7 

5 

9 

2 

9 

5 

9 

9 

9 

9 

9 

Mean 

1.19 

7.00 

1.75 

7.25 

5.56 

7.81 

5.75 

8.56 

7.13 

9.31 

6.31 

8.38 

7.19 

9.06 

Std. 

Dev. 

0.83 

1.67 

1.00 

1.53 

1.82 

1.28 

2.05 

1.03 

1.5 

0.79 

1.92 

1.15 

1.64 

0.93 
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3 . Please rate surface traffic awareness for the following display concepts. 1 = low; 10 = high 

Scenario 1 - arrival / takeoff hold 


EP# 

B 

BMO 

BMOT 

BRIPS 

1 

4 

4 

8 

10 

2 

1 

2 

8 

8 

3 

1 

1 

6 

7 

4 

1 

1 

10 

10 

5 

1 

1 

10 

10 

6 

0 

0 

8 

6 

7 

1 

5 

8 

9 

8 

1 

1 

10 

10 

9 

1 

1 

6 

6 

10 

1 

1 

9 

9 

11 

1 

1 

8 

8 

12 

1 

1 

8 

8 

13 

1 

1 

6 

5 

14 

1 

1 

9 

8 

15 

1 

1 

3 

4 

16 

1 

2 

7 

7 

Mean 

1.13 

1.5 

7.75 

7.81 

Std. Dev 

0.81 

1.26 

1.84 

1.87 


Scenario 2 - departure / intersection departure 


EP# 

B 

BMO 

BMOT 

BRIPS 

1 

4 

4 

9 

10 

2 

1 

2 

8 

9 

3 

1 

1 

6 

6 

4 

1 

1 

9 

9 

5 

1 

1 

8 

10 

6 

0 

0 

8 

6 

7 

1 

5 

9 

8 

8 

1 

1 

10 

10 

9 

1 

3 

6 

8 

10 

1 

1 

9 

9 

11 

1 

1 

8 

8 

12 

1 

1 

6 

6 

13 

1 

3 

7 

6 

14 

1 

1 

6 

5 

15 

1 

1 

4 

5 

16 

1 

2 

5 

5 

Mean 

1.13 

1.75 

7.38 

7.5 

Std. Dev. 

0.81 

1.34 

1.71 

1.90 


53 






Scenario 3 - arrival / departure 


EP# 

B 

BMO 

BMOT 

BRIPS 

1 

4 

4 

10 

10 

2 

1 

2 

9 

10 

3 

1 

1 

6 

6 

4 

1 

1 

8 

8 

5 

0 

0 

8 

8 

6 

0 

3 

8 

6 

7 

1 

6 

8 

9 

8 

10 

10 

10 

10 

9 

1 

1 

4 

4 

10 

1 

1 

9 

9 

11 

1 

1 

8 

8 

12 

1 

1 

5 

6 

13 

1 

1 

5 

5 

14 

1 

1 

3 

3 

15 

1 

1 

4 

4 

16 

1 

1 

5 

5 

Mean 

1.63 

2.19 

6.88 

6.94 

Std. Dev. 

2.39 

2.56 

2.28 

2.35 


Scenario 4 - departure / departure 


EP# 

B 

BMO 

BMOT 

BRIPS 

1 

5 

5 

9 

10 

2 

1 

2 

8 

10 

3 

1 

1 

6 

6 

4 

1 

1 

7 

7 

5 

1 

1 

3 

3 

6 

0 

0 

6 

6 

7 

1 

2 

9 

9 

8 

1 

1 

10 

10 

9 

1 

3 

4 

4 

10 

1 

1 

9 

9 

11 

1 

1 

10 

10 

12 

1 

1 

4 

4 

13 

1 

3 

8 

7 

14 

1 

1 

3 

2 

15 

1 

1 

3 

4 

16 

1 

1 

7 

7 

Mean 

1.19 

1.56 

6.63 

6.75 

Std. Dev. 

1.05 

1.21 

2.55 

2.74 
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Scenario 5 - taxi crossins / departure 


EP# 

B 

BMO 

BMOT 

BRIPS 

1 

5 

5 

10 

10 

2 

1 

3 

9 

10 

3 

1 

1 

6 

7 

4 

1 

1 

8 

7 

5 

1 

1 

10 

10 

6 

0 

0 

10 

8 

7 

1 

4 

9 

8 

8 

1 

1 

10 

10 

9 

1 

1 

8 

8 

10 

1 

1 

9 

9 

11 

1 

1 

8 

8 

12 

1 

1 

6 

6 

13 

1 

3 

6 

5 

14 

1 

1 

8 

7 

15 

1 

1 

2 

2 

16 

1 

1 

9 

9 

Mean 

1.19 

1.63 

8 

7.75 

Std. Dev. 

1.0 5 

1.36 

2.13 

2.14 
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Appendix H: Usability Study Detection Algorithm Evaluation Questionnaire 

Results 


The EPs evaluated the two incursion detection algorithms after each test scenario. The scenarios are 
defined as follows: 


Scenario 1 
Scenario 2 
Scenario 3 
Scenario 4 
Scenario 5 


arrival / takeoff hold 
departure / intersection departure 
arrival / departure 
departure / departure 
taxi crossing / departure 


1. Please rate the effectiveness of the alerting for preventing runway incursions for each display 

concept (based on the runway incursion scenario experienced). 1 = low; 10 = high 

Note: All evaluations were made using the BRIPS display concept (Baseline with perspective map). 



Seenario 1 

Seenario 2 

Seenario 3 

Seenario 4 

Seenario 5 

EP# 

Single 

Stage 

Two 

Stage 

Single 

Stage 

Two 

Stage 

Single 

Stage 

Two 

Stage 

Single 

Stage 

Two 

Stage 

Single 

Stage 

Two 

Stage 

1 

10 

10 

10 

9 

10 

6 

10 

7 

10 

5 

2 

9 

10 

10 

10 

10 

10 

10 

10 

9 

10 

3 

7 

8 

7 

7 

7 


8 

7 

8 

7 

4 

8 

9 

9 

9 

7 

7 

8 

7 

7 

4 

5 

10 

10 

10 

10 

3 

10 

10 

10 

10 

3 

6 

7 

10 

10 

10 

10 

10 

10 

9 

10 

9 

7 

8 

9 

9 

9 

8 

9 

9 

8 

9 

8 

8 

10 

6 

10 

8 

10 

10 

10 

10 

10 

10 

9 

8 

9 

8 

8 

8 


8 

8 

8 

8 

10 

8 

9 

8 

8 

8 

8 

8 

8 

8 

9 

11 

9 

10 

10 

10 

10 

10 

10 

10 

10 

10 

12 

9 

10 

10 

9 

9 


10 

10 

10 

10 

13 

8 

9 

9 

8 

9 


9 

9 

9 

8 

14 

8 

9 

8 

8 

8 

8 

8 

7 

8 

8 

15 

10 

9 

10 

10 

9 


10 

4 

10 

8 

16 

9 

9 

9 

9 

9 


9 

9 

9 

9 

Mean 

8.63 

9.13 

9.19 

8.88 

8.44 

8.8 

9.19 

8.31 

9.06 

7.88 

Std. Dev. 

1.02 

1.02 

0.98 

0.96 

1.79 

1.48 

0.91 

1.66 

1.00 

2.16 


Comments: 

EPl Scenario 2 

EP4 Scenario 1 

EP7 Scenario 1 

Scenario 3 


two stage - alerted later 

two stage - more time, chance to evaluate 

two stage faster 

based on how landed, thought two stage alerted sooner 
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Scenario 4 

single stage faster 


Scenario 5 

single stage faster 

EP9 

Scenario 5 

caution didn’t buy anything 

EP13 

Scenario 5 

caution not beneficial - too close to warning 

EP14 

Scenario 5 

single stage - did not perceive difference in alerting 
two stage - not useful as implemented 

EP15 

Scenario 5 

two stage - no need for two alerts in this case 


2. In general, which type of alerting is better for single pilot general aviation operations? 
Single stage alerting or two stage alerting 

For the choice selected, how much more effective is that alerting type for runway incursion 
prevention? 

1 = low; 10 = high 

Please provide a reason for your answer. 


EP# 

Single 

Stage 

Effectiveness 

Two 

Stage 

Effectiveness 

Reason 

1 

1 




It gave more time to react with single stage in 
sim. I’d rather have two stage alerting with the 
alerting time of single stage. 

2 



1 

2 

Both are excellent, two stage provides a higher 
comfort level. 

3 



1 

4 

It is better to have the caution which seems to 
give you more time to evaluate the situation and 
react accordingly. 

4 

1 

4 

1 

6 

Two stage for approaches only. Single stage 
alerting allowed more time for response during 
ground ops. The two stage, however, allows the 
pilot to know when to start looking on approach 
for other aircraft when concentrating on other 
tasks. 

5 



1 

3 

Seems redundant but a second warning may help 
if the first is not heard. 

6 

1 

8 



With one you first react. With two your mind 
makes you analyze what is happening. 

7 



1 

8 

Two stage seems to provide earlier alerting of 
hazards, enabling pilot attention and decision 
making - proactive responses. Like selective on 
stage in certain cases (takeoff . . .) 

8 

1 

8 



(no comment) 

9 



1 

2 

Earlier notification 

10 



1 

1 

The two stage gave me a chance to look at the 
display prior to the warning. Better SA. 

11 



1 

3 

Single is very good, but two stage offers a little 
more cushion. 

12 

1 

3 



Caution can be confusing 

13 



1 

4 

More time to prepare before acting 
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14 



1 

3 

Early alert for more reaction time 

15 

1 

9 



Late alert for two stage 

16 



1 

4 

Couldn’t tell a big difference 

Total 

6 

6.4 

11 

3.64 



3. With respect to latency, were the incursion alerts provided in a timely manner allowing sufficient 
time to react to the potential conflict? 

All 16 EPs answered YES 

Did either single stage or two stage alerting provide more time for response to the RI? 

Please estimate how much more time the alert provided. 


If not, when should the alert have occurred? 


EP# 

Single 

Stage 

Two Stage 

Seconds 

Comment 

1 

V 


2 


2 


V 

5 to 10 


3 

V 

V 

3 to 4 

Single stage for taxi and departure, two stage for approach 

4 

V 


5 to 10 

Two stage taxi (scenario 5) not enough time. Two stage more time 
on approach. 

5 



Same 

If any thing, alert should have occurred when the other plane 
begins takeoff roll. Two stage for scenario 3 & 5 was late. 

6 

V 


1 to 1.5 

Except for departure, alerts should be sooner. 

7 


V 

1 to 4 

Potentially a lifetime. 

8 



Equal 


9 


V 

2 

Most times single stage is all that is needed. Two stage beneficial 
on approach only. 

10 



Neither 


11 


V 

2 to 3 


12 

V 


1 


13 

V 


3 


14 

V 

V 

2 

Single stage for takeoff, two stage for landing 

15 

V 


3 


16 


V 

10 to 15 

Two stage on approach 

Total 

8 

7 
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Appendix I: Final Semi-Structured Interview Questionnaire Results 


A post-test questionnaire was administered to each EP. The acronyms for the display conditions are 
defined as follows: 


B Baseline 

BMO Baseline with 

BMOT Baseline with 

BA Baseline with 

BAMO Baseline with 

BAMOT Baseline with 

BRIPS Baseline with 


Surface Map and Ownship 
Surface Map and Ownship and Traffic 
Audible Incursion Alerts 

Audible Incursion Alerts, Surface Map and Ownship 
Incursion Alerts, Surface Map and Ownship, Traffic 
Perspective Surface Map 


I. RIPS General Safety 

I. In general, please rate the level of safety felt during runway incursion incidents using the following 
equipage. 1 = Not Safe; 10 = Completely Safe 


EP# 

B 

BMO 

BMOT 

BA 

BAMO 

BAMOT 

BRIPS 

1 

4 

4 

5 

6 

6 

10 

10 

2 

5 

6 

7 

8 

9 

10 

10 

3 

1 

2 

6 

6 

6 

7 

8 

4 

2 

2 

5 

5 

5 

10 

10 

5 

1 

1 

7 

8 

8 

9 

10 

6 

0 

0 

3 

8 

8 

10 

8 

7 

1 

5 

6 

6 

7 

9 

8 

8 

5 

5 

8 

10 

10 

10 

10 

9 

1 

3 

4 

7 

7 

9 

9 

10 

6 

6 

9 

10 

10 

10 

10 

11 

1 

1 

8 

8 

8 

10 

10 

12 

1 

1 

5 

10 

10 

10 

10 

13 

1 

3 

4 

5 

6 

9 

8 

14 

2 

3 

3 

5 

5 

9 

8 

15 

1 

1 

3 

7 

7 

8 

9 

16 

1 

1 

3 

6 

7 

9 

9 

Mean 

2.06 

2.75 

5.38 

7.19 

7.44 

9.31 

9.19 

Std. Dev. 

1.84 

1.95 

2.00 

1.76 

1.67 

0.87 

0.9.1 


59 






II. RIPS General Advantages and Disadvantages 

2a. What do you consider the best feature of the RIPS displays/technology you evaluated today? 


Can the feature be improved? If so, how? 


EP# 

Best Feature 

Improvement 

1 

Showed movement of other aireraft on airport 

Liked perspeetive map better; however, the traffie 
ieons were harder to see 

2 

Provides enhaneed situational awareness, 
ownship loeation, airport environment & traffie 
position & direetion moving 

Capability to vary font size 

3 

Audible alert 

"Warning, traffie 34R" not deseriptive enough 

4 

Airport layout system for increased situational 
awareness 

Text elutter adjustments (ID labels) 

5 

Audio alerts 

Little sooner to mateh the graphie takeoff roll 

6 

Top down view, audible & traffie 

N# no help 

7 

Dynamie and real-time situational awareness 
and hazard warning 

(no eomment) 

8 

N/A 

N/A 

9 

Audible alert 

(no eomment) 

10 

Traffie 

Warning late sometimes / lateney 

11 

Traffie + audible altering 

(no eomment) 

12 

Audible 

(no eomment) 

13 

Combination visual & audible alerts 

Alert a little sooner 

14 

Taxi elearanee 

(no eomment) 

15 

Large red warning symbol with audio alert 

Show ownship with larger symbol, use larger 
traffie symbols at threshold 

16 

Give warning of ineursions when traffie was 
diffieult to see 

(no eomment) 
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2b. What do you consider to be the worst feature of the RIPS displays/technology you evaluated today? 


Can the feature be improved? If so, how? 


EP# 

Worst Feature 

Improvement 

1 

Text warning didn't get my attention as well as 
audible and text 

Maybe use a short flashing box that highlights the 
warning 

2 

Loeation of display in eoekpit - too mueh eye 
movement for eross eheek 

Ability to deelutter ID tags readily 

3 

Top down map looked busy 

Less elutter 

4 

(no eomment) 

(no eomment) 

5 

Did not like to wateh the rotation of the top 
down view 

Not really - leave it in as a ehoiee 

6 

Perspeetive view 

(no eomment) 

7 

No "worst" feature 

Minor tune ups 

8 

N/A 

N/A 

9 

None 

(no eomment) 

10 

(no eomment) 

(no eomment) 

11 

Loeation 

(no eomment) 

12 

Caution ehevron (yellow) hard to see; plaeement 
of display 

Flashing ehevron - reposition panel 

13 

Position of sereen 

Move it 

14 

Clutter, eolor of symbology 

During warning, ehange display to show only 
eonfliet, traffie, speed, distanee, time to eonfliet 

15 

Small symbol for ownship & traffie & delayed 
warning on 2 stage alert 

With 2 stage alert - make warning earlier - 
inerease size of symbols of a/e moving on surfaee 
(traffie and ownship) 

16 

Display a little eluttered in top down display 

(no eomment) 
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2c. What are the advantages and disadvantages of the RIPS displays/technology you evaluated 
compared to the baseline condition (baseline round dials only)? 


EP# 

Advantages 

Disadvantages 

1 

Pin points your location in relation to the other 
a/c on airport and warnings (audible and text) 

No disadvantage 

2 

Situation awareness 

Possible sunlight washout of screen 

3 

Better situational awareness 

Training, more to look at 

4 

Traffic advisories allow for a much higher sense 
of awareness 

The chance a pilot may focus on guidance line 
during approach 

5 

The pilot can see traffic and know their direction 
and location even in poor visibilities and 
distance locations obscured by airport facilities 

Pilots can be too dependent on the technology and 
less on looking out the window. What if some 
plane is not compatible with the technology and is 
not displayed? 

6 

Situational awareness greatly enhanced 

none 

7 

Great improvement in SA and for intuitive and 
information 

Essentially zero 

8 

N/A 

N/A 

9 

Audible + perspective + graphic alert 

None 

10 

Situation awareness 

Would take training to ensure attention 
distribution optimally and how to interpret the 
warnings 

11 

More aware of traffic and surrounding 

Left placement of screen 

12 

Audible alerts + SA 

Audible caution was confusing 

13 

Gives information you need - my position 
relative to traffic & airport features 

(no comment) 

14 

(no comment) 

(no comment) 

15 

Audio alert + red (large) warning lettering & 
symbols on display 

None 

16 

Gave excellent traffic warning especially when 
audio alert used 

None 


III. RIPS Surface Ops Evaluation 

3. Please indicate your level of agreement with each of the following statements for all display 
concepts. 1 = Low; 10 = High 

3a. Where am I? The display concept provides sufficient awareness of my ownship position with 
respect to runways, taxiways, and stationary objects. 

If unequal rating, please indicate reason. 

If not “completely agree” for Surface Map conditions, what can be done to improve the concept to 
increase awareness? 
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EP# 

B 

BMO 

BMOT 

BA 

BAMO 

BAMOT 

BRIPS 

Reasons 

Improvement 

1 

2 

10 

10 

2 

10 

10 

10 

Baseline doesn't 
show my 
orientation on 
airport 

(no eomment) 

2 

1 

10 

10 

1 

10 

10 

10 

(no eomment) 

(no eomment) 

3 

1 

6 

6 

1 

6 

6 

7 

(no eomment) 

(no eomment) 

4 

2 

8 

8 

2 

8 

8 

8 

(no eomment) 

No vertieal 
indieations 

5 

1 

10 

10 

1 

10 

10 

10 

(no eomment) 

(no eomment) 

6 

1 

8 

8 

1 

8 

10 

8 

Easier to 
interpret top 
down 

(no eomment) 

7 

1 

10 

10 

2 

10 

10 

9 

Perspeetive 
slightly poor due 
to relative 
visibility of 
eertain areas 

Same eomment 

8 

6 

8 

8 

6 

8 

10 

10 

(no eomment) 

(no eomment) 

9 

1 

9 

9 

1 

9 

9 

9 

Audible provides 
no info for my 
position 
awareness 

Nothing 

10 

4 

4 

7 

6 

8 

10 

10 

Airport 
environment + 
traffie + alerting 

N/A 

11 

1 

8 

8 

2 

8 

8 

10 

Perspeetive view 

(no eomment) 

12 

1 

10 

10 

1 

10 

10 

10 

Surfaee map 
provides all the 
information you 
need 

(no eomment) 

13 

1 

8 

9 

1 

9 

9 

8 

Prefer top down 
view on airport 
surfaee; traffie 
alert without 
map position not 
as helpful 

Move display; 
adjust seale 

14 

1 

8 

8 

1 

8 

8 

8 

(no eomment) 

(no eomment) 

15 

1 

10 

10 

2 

10 

10 

10 

(no eomment) 

(no eomment) 

16 

1 

9 

9 

1 

9 

9 

9 

(no eomment) 

Draw pieture 
of elearanee 

Mean 

1.63 

8.5 

8.75 

1.94 

8.81 

9.19 

9.125 


Std. 

Dev. 

1.41 

1.67 

1.24 

1.65 

1.17 

1.17 

1.02 


3b. Where am I relative to other movins objects? The display concept provides sufficient awareness of 
my ownship position with respect to moving traffic, such as vehicles and other aircraft. 


If unequal rating, please indicate reason. 
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If not “completely agree” for surface map with traffic conditions, what can be done to improve the 
concept to increase awareness? 


EP# 

B 

BMO 

BMOT 

BA 

BAMO 

BAMOT 

BRIPS 

Reasons 

Improvement 

1 

1 

1 

9 

1 

1 

9 

10 

No traffic info 
without map 
(baseline rounds 
dials only) 

Perspective 
view gave a 
broader view of 
airport traffic 

2 

1 

1 

8 

5 

5 

10 

10 

(no comment) 

Audible/ visual 
alerting 

enhances SA to 
moving traffic 

3 

2 

2 

6 

2 

2 

6 

7 

(no comment) 

(no comment) 

4 

1 

1 

8 

3 

3 

9 

9 

Alert brings any 
incursions to the 
users notice 

(no comment) 

5 

1 

1 

10 

2 

2 

10 

10 

Audible alerting 
provides info for 
one a/c not all 

(no comment) 

6 

0 

0 

8 

6 

6 

10 

8 

Like top down 

(no comment) 

7 

1 

2 

8 

2 

2 

10 

9 

Need to see traffic 
+ self on map 

(no comment) 

8 

4 

4 

8 

6 

6 

10 

10 

(no comment) 

(no comment) 

9 

1 

1 

6 

1 

1 

8 

8 

Msg & audible 
alerts adds value on 
other traffic 
position and need 
to respond 

(no comment) 

10 

4 

4 

7 

6 

6 

10 

10 

Elements are 
shown on map + 
relevant features, 
identifiable 
landmarks, etc. 

(no comment) 

11 

5 

5 

8 

7 

7 

10 

10 

(no comment) 

(no comment) 

12 

1 

1 

10 

5 

5 

10 

10 

(no comment) 

(no comment) 

13 

1 

3 

8 

6 

6 

9 

8 

Prefer top down 

Position of 
display; change 
scale 

14 

1 

1 

8 

5 

5 

9 

9 

(no comment) 

(no comment) 

15 

1 

1 

10 

1 

1 

10 

10 

The Ts don't show 
traffic 

(no comment) 

16 

1 

1 

9 

7 

9 

9 

9 

(no comment) 

If unit could 
show a potential 
conflict in a 
different color 

Mean 

1.63 

1.81 

8.19 

4.06 

4.25 

9.31 

9.19 


Std. 

Dev. 

1.41 

1.42 

1.22 

2.26 

2.54 

1.08 

0.98 
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IV, RIPS Display and Alerts 


4. Please indicate the effectiveness of the perspective map display compared to the top-down views for 
prevention of runway incursion. 1 = Low; 10= High 


Reason for rating. 


EP# 

Rating 

Reason 

1 

9 

The perspeetive gave a broader view of airport traffie 

2 

2 

Preferenee 

3 

4 

Easier to piek out ownship in less time 

4 

0 

No differenee 

5 

5 

Top down showed more, perspeetive was easier to look at 

6 

2 

Fast orientation to top down 

7 

8 

Foeus on approaeh environment (for approaehes only, perspeetive more effeetive) 

8 

5 

(no eomment) 

9 

1 

Same information and equally as readable 

10 

0 

Same effeetiveness; preferenee is for 2D top down 

11 

1 

(no eomment) 

12 

3 

Top down was easier to interpret, perspeetive items get lost 

13 

3 

Top down gives broader view to sides & rear 

14 

1 

(no eomment) 

15 

0 

No differenee - both maps displayed the alert warnings satisfaetorily 

16 

2 

Map less eluttered 

Mean 

2.88 


Std. Dev. 

2.73 


5. When traffic was presented on surface map, was incurison traffic easily discernable? 
Yes or No 

All 16 EPs responded Yes. 


If yes, please indicate how discernable. 1 = Low, 10 = High 


EP 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 


10 

10 

7.5 

10 

9 

8 

10 

5 

9 

9 

6 

8 

7.5 

5 

10 

10 

Mean 

8.38 


Std. Dev. 

1.77 


6. Is traffic presentation necessary on surface map to prevent runway incursions? 
Yes or No 

EPs 8 and 15 responded No. The other 14 EPs responded Yes. 
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If so, please indicate how much additional situation awareness is provided with the addition of traffic 
presentation on the surface map. 


Comparing surface map with ownship TO surface map with ownship and traffic, situation awareness 
enhancement is: 1 = Low; 10 = High 


EP 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 


10 

9 

8 

7 

10 

8 

9 

4 

10 

8 

8 

10 

10 

8 

9 

9 

Mean 

8.56 


Std. Dev. 

1.56 


7. Is graphical presentation of alerts necessary on surface map to prevent runway incursions? 
Yes or No 


If so, please indicate how much additional situation awareness is provided with the addition of 
graphical alerting on the surface map, 1 = Low; 10 = High 


EP# 

Yes 

No 

Rating 

Comment 

1 

V 


10 


2 


a/ 


Graphie not needed if audible alert available 

3 

V 


3 


4 


a/ 


But is helpful 

5 

V 


9 


6 

V 


8 


7 

V 


9 


8 


a/ 

3 


9 


a/ 



10 

V 


8 


11 

V 


3 


12 

a/ 


4 


13 

a/ 


8 


14 

V 


4 


15 


V 



16 


V 



Total 

10 

6 

6.27 

Mean 




2.83 

Std. Dev. 


8. If a runway incursion situation occurs, which of the following is most likely to bring the event to 
your attention? If multiple, please indicate the rank order. 

OTW = out-the-window 
Map = surface map 
Audible = audible alerting 

Note: For B and BMO, OTW is the only method of acquiring incursion traffic. 
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EP# 

BMOT/ 

OTW 

BMOT/ 

map 

BA / OTW 

BA/ 

audible 

BMO/ 

OTW 

BMO/ 

audible 

BAMOT/ 

OTW 

BAMOT/ 

audible 

BAMOT/ 

map 

BRIPS / 
OTW 

BRIPS / 
audible 

BRIPS / 
map 

1 

2 

1 

1 

2 

1 

2 

2 

3 

1 

2 

3 

1 

2 

1 

2 

2 

1 

2 

1 

2 

1 

3 

2 

1 

3 

3 

2 

1 

2 

1 

2 

1 

3 

1 

2 

3 

1 

2 

4 

2 

1 

1 

2 

1 

2 

2 

3 

1 

2 

3 

1 

5 

2 

1 

1 

2 

1 

2 

2 

3 

1 

2 

3 

1 

6 

1 

2 

2 

1 

2 

1 

2 

1 

3 

2 

1 

3 

7 

2 

1 

2 

1 

2 

1 

3 

1 

2 

3 

1 

2 

8 

1 

2 

1 

2 

1 

2 

1 

2 

3 

1 

2 

3 

9 

2 

1 

2 

1 

2 

1 

3 

1 

2 

3 

1 

2 

10 

1 

2 

2 

1 

2 

1 

3 

1 

2 

3 

1 

2 

11 

1 

2 

1 

2 

1 

2 

3 

2 

1 

3 

2 

1 

12 

1 

2 

2 

1 

2 

1 

2 

1 

3 

2 

1 

3 

13 

2 

1 

2 

1 

2 

1 

3 

1 

2 

3 

1 

2 

14 

1 

2 

1 

2 

1 

2 

1 

2 

3 

1 

2 

3 

15 

1 

2 

2 

1 

2 

1 

3 

1 

2 

3 

1 

2 

16 

2 

1 

2 

1 

2 

1 

3 

1 

2 

3 

1 

2 

Mean 

1.5 

1.5 

1.63 

1.38 

1.63 

1.38 

2.38 

1.56 

2.06 

2.38 

1.56 

2.06 

Std. 

Dev. 

0.52 

0.52 

0.5 

0.5 

0.5 

0.5 

0.72 

0.81 

0.77 

0.72 

0.81 

0.77 


9 . Upon receiving an incursion alert, was your immediate reaction to: 
Confirm the hazard, Take evasive action, or Other 


Please give a reason for your answer. 


EP# 

Confirm 

Evasive Aetion 

Other 

Reason 

1 


V 


I saw traffie first and the audible eonfirmed traffie ineursion 
and I took aetion. 

2 


V 


Limited time to reaet 

3 


V 


(no eomment) 

4 

V 



Evaluate what the evasive aetion should be 

5 


V 


The audio alerts were given after I saw the ineursion 

6 


V 


Training that warning requires aetion 

7 

V 

V 


Initiate immediately and eonfirm, then deeide 

8 


V 


(no eomment) 

9 


V 


Need to aet quiekly 

10 


V 


Wasn’t in an emergeney situation so was safe to get to safe 
altitude 
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11 


V 


Best course of action 

12 


V 


Trusted the alert 

13 

V 



I’m a skeptic 

14 


V 


(no comment) 

15 


V 


At slow speeds best to abort 

16 

V 



May miss an approach or create a dangerous situation if 
hazard is not confirmed first 

Total 

4 

13 

0 



What do you feel is the appropriate procedure on receipt of a RI alert for the following? 
Landing, Takeoff and Taxi 


EP# 

Landing 

Takeoff 

Taxi 

1 

Go around 

Abort 

Stop 

2 

Go around 


Stop 

3 

Go around 

Abort 

Stop 

4 

Go around 

Abort 

Stop 

5 

Go around 

Abort 

Stop 

6 

Go around 

Abort 

Stop 

7 

Go around 

Abort 

Stop 

8 

Go around 

Abort 

Stop 

9 

Go around 

Abort 

Stop 

10 

Go around 

Abort 

Stop 

11 

Go around 

Abort 

Stop 

12 

Go around 

Abort 

Stop 

13 

Go around 


Stop 

14 

Go around 

Contention 

Stop 

15 

Go around 

Abort 

Stop 

16 

Go around 

Abort 

Stop 


10. Which display combination would provide a minimal configuration to be effective for runway 
incursion prevention. 


EP# 

Audible Alert 

Surface Map with 
Ownship only 

Surface Map with 
Ownship & Traffic 

Baseline Round Dials 

1 



V 


2 



V 


3 

V 




4 

V 




5 



V 


6 

V 




7 

V 


V 


8 

V 
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For an optimal configuration? 


EP# 

Audible Alert 

Surface Map with 
Ownship only 

Surface Map with 
Ownship & Traffic 

Baseline Round Dials 

1 

V 


V 


2 

V 


V 


3 

V 


V 


4 

V 


V 


5 

V 


V 


6 

V 


V 


7 

V 


V 


8 

V 


V 


9 

V 


V 


10 

V 


V 


11 

V 


V 


12 

V 


V 


13 

V 


V 


14 

V 


V 


15 

V 


V 


16 

V 


V 


Total 

16 

0 

16 

0 


11 . Please rate the effectiveness of the terms used for the incursion alerts. 1 = Low; 10 = High 


EP# 

“Warning, 
traffic on 
RWY” 

“Caution, 
traffic on 
RWY” 

“ Warning, 
traffic departing 
RWY” 

“Caution, traffic 
departing 
RWY” 

“Warning, 

traffic 

approaching 

RWY” 

“Caution, 

traffic 

approaching 

RWY” 

1 

10 

10 

10 

10 

10 

10 

2 

10 

10 

10 

10 

10 

10 

3 

5 

5 

8 

8 

8 

8 

4 

10 

10 

10 

10 

10 

10 
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5 

9 

9 

9 

9 

N/A 

N/A 

6 

10 

7 

10 

7 

10 

7 

7 

9 

9 

9 

9 

9 

9 

8 

10 

10 

10 

10 

10 

10 

9 

9 

9 

9 

9 

9 

9 

10 

10 

10 

10 

10 

10 

10 

11 

10 

10 

10 

10 

10 

10 

12 

10 

5 

10 

5 

10 

5 

13 

9 

9 

9 

9 

9 

9 

14 

8 

8 

8 

8 

8 

8 

15 

8 

4 

4 

8 

3 

3 

16 

10 

10 

10 

10 

10 

10 

Mean 

9.25 

8.44 

9.13 

8.88 

9.07 

8.53 

Std. Dev. 

1.29 

2.06 

1.54 

1.41 

1.83 

2.10 


If not “completely effective”, would another term be more effective or descriptive? 
How could the alert term be improved (i.e.wording)? 

The following EPs responded to this question: 

EP 3 More specific about location of incursion traffic 

EP 12 Delete cautions 

EP 13 Repeat warning - may not be understood first time 
EP 1 5 Caution traffic crossing left, to right, right to left 


12a. For single pilot operations, was the two stage alerting (“Caution” & “Warning”) more effective in 
preventing runway incidents than single stage alerting (“Warning” only) ? Yes or No 


12b. Did the two stage alerting (“Caution” <6 “Warning”) enhance your situation awareness (SA) 
more than the single stage alerting (“Warning” only)? Yes or No 


12c. If yes, please indicate how much more the two stage alert enhanced your SA compared to the one 
stage alert 1 = Minimal; 10 = Substantially 

What was the reason for your rating? 

What are the benefits, if any, of two stage alerting? 


70 






12a 

12b 

12e 



EP# 

Yes 

No 

Yes 

No 


Reason 

Benefits 

1 


V 


V 



Gives the pilot time to reaet 
to threat of ineursion 

2 


V 

V 


4 

Inereased eomfort level with 
having more pereeived time for 
deeision making 


3 

V 


V 


4 

Felt like I had more time 

Reaetion time, seene 
evaluation 

4 


V 

V 


4.5 

Allowed more time for response 
and reaetion when giving a 
eaution and warning 

Same eomment 

5 


V 


V 



Only needed if the first alert 
is not heard 

6 


V 


V 



Better that none, single more 
effeetive 

7 

V 


V 


8 

Draws attention to hazard earlier 
& provides assessment + 
deeision time advantage 

Same eomment 

8 


V 


V 




9 

V 


V 


2 

Very slight improvement in 
timing of information 

Personal option to aet sooner 

10 

V 


V 


4 

Gave opportunity to look at 
surfaee map to eonfirm 

More tim to analyze 
situation 

11 

V 


V 


4.5 

Provided more response time 

Same Comment 

12 


V 


V 


Caution was eonfusing 

N/A 

13 


V 

V 


1.5 

Not a big differenee 

More time to eonfirm / 
eonsider 

14 

V 


V 


7 

Single for takeoff; two stage for 
landing 

Early warning 

15 


V 


V 



None 

16 


V 


V 




Total 

6 

10 

9 

7 

4.39 

Mean 



2.07 

Std. Dev. 
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13 . For each display concept, indicate which type of alerting provides the greatest amount of runway 
incursion prevention awareness and the amount of situation awareness enhancement (based on the 
choice you selected) for the following displays: 1 = Minimal; 10 = Substantially 


S = Single stage alert T = Two stage alert SA = Situation awareness enhancement 


EP# 

BA + S 

< 

BA + T 

< 

CO 

BAMO + S 

< 

CO 

BAMO + T 

< 

CO 

BAMOT + S 

< 

CO 

BAMOT + T 

< 

CO 

BRIPS + S 

< 

CO 

BRIPS + T 

< 

CO 

1 

1 

8 



1 

8 



1 

10 



1 

10 



2 



1 

6 



1 

7 



1 

9 



1 

9 

3 



1 

5 



1 

5 



1 

7 



1 

8 

4 



1 

5 



1 

5 

1 

9 



1 

9 



5 



1 

5 



1 

5 



1 

9 



1 

10 

6 

1 

8 



1 

8 



1 

10 



1 

8 



7 



1 

8 



1 

8 



1 

8 



1 

8 

8 

1 

4 



1 

4 



1 

4 



1 

4 



9 



1 

2 



1 

2 



1 

3 



1 

3 

10 



1 

3 



1 

3 



1 

3 



1 

3 

11 



1 

4 



1 

4 



1 

2 



1 

2 

12 

1 

10 



1 

10 



1 

10 



1 

10 



13 



1 

2 



1 

2 



1 

2 



1 

2 

14 



1 

2 



1 

2 



1 

2 



1 

2 

15 

1 

7 



1 

8 



1 

9 



1 

10 



16 

1 

2 



1 

2 



1 

2 



1 

2 



Total 

6 


10 


6 


10 


7 


9 


7 


9 


Mean 


6.5 


4.2 


6.7 


4.3 


7.7 


5.0 


7.6 


5.2 

Std. 

Dev. 


2.95 


1.99 


3.01 


2.11 


3.30 


3.16 


3.23 


3.42 
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14. Would you like to be given escape guidance on the maneuver to take to avoid the conflict in 
addition to the runway incursion alert? 



On Final 
Approach 

On Takeoff 
Roll 

When 
Taxiing 
Across a 
Runway? 

What type of advisory / guidance would you like to see 
during landing, during takeoff, or during taxi across a 
runway? 

EP# 

Yes 

No 

Yes 

No 

Yes 

No 

1 

V 



V 


V 

Safest maneuver to use to avoid collision 

2 

V 



V 


V 

Missed approach instructions 

3 

V 


V 


V 


Simple escape guidance (e.g. Climb right, descend left) 

4 

V 



V 


V 

Guidance by heading to go behind the conflicting aircraft 

5 

V 


V 


V 


Landing: direction to go around; takeoff: “abort takeoff’; 
taxi: “stop taxi” 

6 

V 



V 


V 

Go around or missed approach, climb & steering 
commands 

7 

V 


V 


V 


Whatever appears to provide relatively maximal 
probability of incident avoidance, resolution of imminent 
hazards, and aircraft/crew survival 

8 


V 


V 


V 

If advising, simple command: “hold short” 

9 

V 


V 


V 


Landing: course & climb decision; takeoff: direction 
change if needed; taxi: just stop 

10 

V 


V 


V 


Explicit aural annunciation and display 

11 


V 


V 

V 


Stop! 

12 


V 


V 


V 


13 

V 


V 


V 


If at controlled airport, issue vector, clearance etc.; if at 
uncontrolled field, short statement of bese evasive action 

14 

V 


V 


V 



15 


V 


V 


V 


16 

V 


V 


V 


Direction - stop - turn - heading? Altitude? 

Total 

12 

4 

8 

8 

9 

7 



15. Please provide any additional comments that will help us in our evaluation and development of the 
synthetic vision system concepts. 

The following EPs responded to this question: 

EP 1 Increase volume on the aircraft noise in cockpit. 

EP 4 Velocity vector display on map 

EP 9 Value of surface map may have increased if in a different location, closer to primary displays 

in multifunction display area 
EP 1 3 Why not put traffic lights at hold lines 

EP 1 5 Eocation of display needs to be within pilots heads-up field of view; land-and- ho Id-short 

planning for all cross-runway landings 
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Appendix J: Paired Comparisons Questionnaire 

Overview: This questionnaire is designed to allow statistical analysis of your subjective assessment of 
situation awareness and runway incursion prevention for each of the following display configurations you 
evaluated today. Please look at the pictures of the display concepts when making your comparisons. 

Paired Comparison Rating Instructions: Each paired comparison will be listed on the left side of the 
questionnaire. You will be asked to make separate comparisons based on what you experienced during 
the flying tasks in the simulator. 

Situation Awareness: If situation awareness is not equal, indicate the magnitude of the difference by 
marking the appropriate box on the scale to the right of the comparison. The following definition of 
situation awareness should be used for reference: 

The pilot ’s awareness and understanding of all factors that will contribute to the safe flying of 
their aircraft under normal and non-normal conditions. 

Runway Incursion Prevention Awareness: If runway incursion awareness is not equal, indicate the 
magnitude of the difference by marking the appropriate box on the scale to the right of the comparison. 
The following definition should be used for reference: 

The pilot ’s awareness and understanding of the dynamic environment and degree to which he or 
she is aware of and can successfully avoid, potential incursions with other aircraft or vehicles 
in the airport environment. 

The following example shows how to make the comparisons. Do not take an excessive amount of time 
on each comparison; your first impression is usually best. However, please feel free to correct any 
comparisons. Also, the data will be checked for consistency; if the results are inconsistent, you may be 
asked to clarify your responses. 


Situation Awareness Comparisons 

If not equal, how much more or how much less? 
Barely Substantially 

Display Concept ‘X’ 


Provides ( V more)( equal)( less) SA than 

V 

Display Concept ‘Y’ 
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Situation Awareness Comparisons 


Baseline round dials only (B) 

Provides ( more)( equal)( less) SA than 
B + surface map with ownship only (BMO) 


Baseline round dials only (B) 

Provides ( more)( equal)( less) SA than 
B + surface map with ownship + traffic (BMOT) 


Baseline round dials only (B) 

Provides ( more)( equal)( less) SA than 

B + audible alerting only (BA) 


Baseline round dials only (B) 

Provides ( more)( equal)( less) SA than 

B + audible alerting + map with ownship only 
(BAMO) 


Baseline round dials only (B) 

Provides ( more)( equal)( less) SA than 
B + audible & graphical alerting + map with 
ownship + traffic (BAMOT) 


Baseline round dials only (B) 

Provides ( more)( equal)( less) SA than 
B + audible & graphical alerting + perspective 
map (BRIPS) 


B + surface map with ownship only (BMO) 

Provides ( more)( equal)( less) SA than 


B + surface map with ownship + traffic (BMOT) 


B + surface map with ownship only (BMO) 

Provides ( more)( equal)( less) SA than 


B + audible alerting only (BA) 


B + surface map with ownship only (BMO) 

Provides ( more)( equal)( less) SA than 
B + audible alerting + map with ownship only 
(BAMO) 


B + surface map with ownship only (BMO) 

Provides ( more)( equal)( less) SA than 


B + audible & graphical alerting + map with 
ownship + traffic (BAMOT) 


B + surface map with ownship only (BMO) 

Provides ( more)( equal)( less) SA than 
B + audible & graphical alerting +perspective 
map (BRIPS) 


If mh equal, how much more or how much less? 
Barely Substantially 
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Situation Awareness Comparisons 


B + surface map with ownship + traffic (BMOT) 
Provides ( more)( equal)( less) SA than 
B + audible alerting only (BA) 


B + surface map with ownship + traffic (BMOT) 
Provides ( more)( equal)( less) SA than 
B + audible alerting + map with ownship only 
(BAMO) 


B + surface map with ownship + traffic (BMOT) 
Provides ( more)( equal)( less) SA than 
B + audible & graphical alerting + map with 
ownship + traffic (BAMOT) 


B + surface map with ownship + traffic (BMOT) 
Provides ( more)( equal)( less) SA than 
B + audible & graphical alerting + perspective 
map (BRIPS) 


If not equal, how much more or how much less? 
Barely Substantially 


B + audible alerting only (BA) 


Provides ( more)( equal)( less) SA than 


B + audible alerting + map with ownship only 
(BAMO) 


B + audible alerting only (BA) 


Provides ( more)( equal)( less) SA than 


B + audible & graphical alerting + map with 
ownship + traffic (BAMOT) 


B + audible alerting only (BA) 


Provides ( more)( equal)( less) SA than 


B + audible & graphical alerting +perspective 
map (BRIPS) 



B + audible alerting + map with ownship only 
(BAMO) 


Provides ( more)( equal)( less) SA than 


B + audible & graphical alerting + map with 
ownship + traffic (BAMOT) 


B + audible alerting + map with ownship only 
(BAMO) 


Provides ( more)( equal)( less) SA than 


B + audible & graphical alerting +perspective 
map (BRIPS) 



B + audible alerting + map with ownship + 
traffic (BAMOT) 


Provides ( more)( equal)( less) SA than 


B + audible & graphical alerting +perspective 
map (BRIPS) 
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Runway Incursion Prevention if not equal, how much more or how much less? 

Barely Substantially 

Baseline round dials only (B) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + surface map + ownship only (BMP) 

Baseline round dials only (B) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + surface map + ownship + traffic (BMOT) 

Baseline round dials only (B) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible alerting only (BA) 

Baseline round dials only (B) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible alerting + map + ownship (BAMO) 

Baseline round dials only (B) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible & graphical alerting + map 

+ ownship + traffic (BAMOT) 

Baseline round dials only (B) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible & graphical alerting + perspective 
map (BRIPS) 

B + surface map + ownship only (BMP) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + surface map + ownship + traffic (BMOT) 

B + surface map + ownship only (BMP) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible alerting only (BA) 

B + surface map + ownship only (BMP) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible alerting + map + ownship (BAMO) 

B + surface map + ownship only (BMP) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible & graphical alerting + map + 

ownship + traffic (BAMOT) 

B + surface map with ownship only (BMP) 

Provides ( more)( equal)( less) potential for 

RI prevention than 
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Runway Incursion Prevention if not equal, how much more or how much less? 

Barely Substantially 

B + surface map + ownship + traffic (BMOT) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible alerting only (BA) 

B + surface map + ownship + traffic (BMOT) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible alerting + map + ownship (BAMO) 

B + surface map + ownship + traffic (BMOT) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible & graphical alerting + map with 
ownship + traffic (BAMOT) 

B + surface map + ownship + traffic (BMOT) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible & graphical alerting + perspective 
map (BRIPS) 

B + audible alerting only (BA) 

Provides ( more)( equal)( less) potential for RI 

prevention than 

B + audible alerting + map + ownship (BAMO) 

B + audible alerting only (BA) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible & graphical alerting + map + 

ownship + traffic (BAMOT) 

B + audible alerting only (BA) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible & graphical alerting +perspective 
map (BRIPS) 

B + audible alerting + map + ownship (BAMO) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible & graphical alerting + map + 

ownship + traffic (BAMOT) 

B + audible alerting + map + ownship only 

(BAMO) 

Provides ( more)( equal)( less) potential for 

RI prevention than 

B + audible & graphical alerting +perspective 

map (BRIPS) 
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B + audible alerting + map + ownship + traffic 
(BAMOT) 


Provides ( more)( equal)( less) potential for 

R1 prevention than 


B + audible & graphical alerting +perspective 
map (BRIPS) 
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